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Advanced  Technology  for 
Portable  Personal  Visualization 


1.0  Summary 

De^ite  the  recent  avalanche  of  publicity  on  Virtual  Reality,  die  state  of  the  an  in  head-mounted  displays 
is  s^  at  die  "toy"  st^e.  Although  extravagant  claims  are  made  almost  daily  on  TV  and  in  the  popdar 
press,  Acre  is  very  lide  that  can  be  usefully  accomplished  today  with  \^nual  Reality  technolo^es. 

,  We  at  WC  have  been  working  since  the  early  1970's  on  aspects  of  these  technologies  and  have  been 
advancing  the  state  of  the  an  by  a  "driving  problem"  approach;  we  let  the  needs  of  selected  applications 
stimulate  the  directitm  of  the  technologic^  developments,  and  then  test  new  results  by  their  impact  on 
solving  the  origir^  application.  We  have  been  working  on  three  application  areas:  molecular  modeling, 
3D  m^cal  imaging,  and  modeling  of  architectural  interiors.^) 

^  We  have  put  together  complete  systems  that  include  the  head-mounted  display  device,  the  display 
generation  hardware,  and  the  he^  and  hand  tracker.  We  bought  the  comi^nents  when  available  and 
built  the  components  when  there  was  a  clear  advantage  to  doing  so: 

a)  We  built,  under  separate  major  funding  by  DARPA  and  NSF,  the  display  generation  hardware  (our 

most  recent  machine  is  Pixel-Planes  5),  -r 

b)  We  budt  an  electro-0)tical  helmet-tracing  system  (tire  "ceiling  tracker")  that  can  determine  position 
..  and  orientation  over  a  large  area  (currently  under  a  10  x  12  foot  ceiling)  —  to  our  knowledge,  the 

first  deax}nstrated  scalable-area  helmet  tracker. 

c)  We  bniy  head-mounted  display  devices,  but  ccxitinue  to  build  bead-mounted  display  devices  with 

"see-through"  capability  -  the  ability  for  the  wearer  to  observe  virtual  world  objects  superimposed 
onto  his/her  physical  surroundings.  _ 

Our  major  accomplishment  in  this  period  was  a  week  of  comprehensive  denx>nstrations  at 
SIGGRAPH’91,  the  world’s  premier  computer  graphics  conference  (Las  Vegas,  July  28  -  August  2, 
1991).  In  the  juried  Tomorrow’s  Realities  Gallery,  the  UNC  group  was  allocated  the  largest  area  of  any 
exhittitor  (28  x  42  feet),  and  was  approved  for  the  largest  number  of  distina  demonstrations  (7): 

1)  Moving  about  the  interior  of  a  virtual  building  while  walking  under  our  custom  10  x  12-foot  ceiling, 

2)  Using  a  treadmill  with  steerable  bicycle  handlebars  to  walk  through  larger  areas  of  a  virtual 

building, 

3)  "Flying"  through  a  space  filled  with  various  models  of  large  molecules, 

4)  Bicycling  through  an  outdoor  environment  cxi  a  bike  whose  pedals  exen  forces  depending  on  the 

steepness  of  the  terrain, 

5)  "Fl)^g"  about  an  outdexx*  scene  that  included  realistically  textured  clouds  and  terrain, 

6)  Designing  the  configuration  of  multiple  radiation  therapy  beams  to  f(x;us  onto  a  tumor  nestled  inside 

a  patient  oxxlel  with  multiple  (heal Ay)  major  organs, 

7)  "Fluting"  around  a  virtual  world  in  which  a  second,  indq>endently  moving,  user  is  modeling  3D 

objects  (feels  like  being  immersed  in  a  3D  MacDraw  world). 

All  these  demonstrations  were  for  individual  users;  Ae  guest  wore  a  stereo,  tracked  head  gear  and 
Ateracted  wi A  Ae  system  for  about  4  minutes.  Crowds  around  Ae  booA  area  observed  on  overhead 
monitors  Ae  images  that  Ae  individual  saw  m  the  head  gear. 
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1.1  Goals  of  the  Head-Mounted  Display  Project 

•  Demonstrate  the  usefulness  of  the  head-mounted  display  (HMD)  in  real  applications. 

•  Improve  the  hardware  subsystems  which  currently  limit  performance  of  HMD  (tracker,  HMD  optics  and 
display,  real-dme  graphics  generation). 

•  Design  arrd  inclement  software  base  to  support  HMD  functions. 

•  Integrate  visual,  auditory  and  haptic  (force  feedback)  displays  into  a  working  system. 

•  Build  new  input  devices  and  investigate  mediods  cf  manual  control  suitable  to  a  head-mounted  display. 


1.2  Goals  of  the  Optical  Tracker  Subproject 

•  Develop  long-range  trackers  for  head-mounted  displays.  For  the  short  term  develop  a  system  that  can 
track  an  HMD  inside  a  room-sized  environment  the  long  term  explore  technologies  that  have 
unlimited  range  without  requiring  modificatitxi  of  the  environment 

•  Work  on  improving  the  other  aspects  of  HMD  tracking:  latency,  speed,  accuracy,  and  resolution. 


2.0  Summary  of  Major  Accomplishments 

•  The  UNC  systems  (including  the  ceiling  tracker)  were  successfully  demonstrated  with  seven  distinct 
Implications  in  the  Tooxnrow's  Realities  Gallery  at  the  SIGGRAPH  ‘91  conference  in  Las  Vegas. 

2.1  Head-Mounted  Display  System 

•  We  demonstrated  a  prototype  system  integrating  ultrasound  scanner  input  into  a  virtual  world  in  the 
Tomorrow's  Realities  GaUery  at  the  SIGGRAPH  '91  conference. 

•  We  demonstrated  a  prototype  system  integrating  monocular  live  video  with  opaque  HMD  in  the 
Tomorrow's  Realities  GaUery  at  the  SIGGRAPH  '91  conference.. 

•  We  have  ported  aU  major  HMD  appUcations  from  Pixel-Planes  4  to  Pixel-Planes  5. 

•  The  main  libraries  for  virtual-worlds  applications  now  suppon  ^pUcations  on  either  host  with  minimal 
changes. 

•  Adlib,  a  library  for  handling  analog-digital  devices  in  a  transparent  maimer  has  been  created  and  fuUy 
documented. 

•  SoundUb,  a  library  far  playing  sounds  on  the  Macintosh  has  been  created  and  documented.  Also,  we 
acquired  a  lilvary  of  more  than  1,(X)0  sounds  for  use  with  the  sound  Macintosh. 

•  AU  HMD-based  programs  now  use  the  new  base  software  with  vUb,  trackerUb,  and  adUb. 

•  SdiUb,  a  serial-device  interface  Ubrary,  has  been  created  and  documented.  SdUib  aUows  applications  to 
communicate  with  serial  devices  across  die  network  as  if  they  were  connected  to  the  user’s  machine. 
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•  W6  have  created  many  new  v^nual-worlds  applications.  Among  them  are; 

•  'V<Tn‘  a  three-dimensional  modeler.  This  t^licadon  grew  out  of  a  software  engineering  class  project 
and  is  now  a  fiilly  documented  tool  for  creating  three-dimensicMial  models  from  within  the  virtual 
world. 

-  Vixen:  a  model-viewing  program.  This  application  is  also  fully  documented  as  a  general-purpose  tool 
and  allows  the  user  to  view  any  3D  model,  grab  it,  scale  it,  and  fly  around  it. 

.  Gallery:  a  tool  for  creating  and  exploring  interactive  environments  rapidly  and  easily.  It  can  be  used 
to  do  rapid  prototyping  of  interaction  metiiods  and  world  layout  or  to  create  interesting  worlds  to 
explore. 

•  Rtpdenx):  a  program  for  demonstrating  the  placement  of  radiation  beams  for  cancer  therapy.  Related 
to  James  Qiung’s  disMartation  work. 

•  Viper  an  adaptation  of  vixen  designed  to  deliver  near-real-time  performance.  This  application 
monitors  its  update  rate  and  simplifies  the  scene  until  the  requested  rate  is  achieved.  Users  can  press  a 
button  to  request  a  full-detail  view  of  the  scene  <xice  they  have  found  the  area  in  which  they  are 
interested. 

•  We  integrated  the  new  ceiling  tracker  and  associated  HMD  into  the  system;  any  existing  £q>plication  can 
run  under  the  UNC  ceiling  tracker  as  if  it  were  just  another  tracker/HMD  ccxnbination. 

•  We  integrated  the  force-feedback  ARM  into  trackeilib  so  tiiat  it  can  be  used  as  a  tracker. 

•  All  virtual-worlds  applications  can  now  use  sound,  visuals,  and  force  feedback. 

•  We  have  done  and  evaluated  user  studies  to  explcnv  different  modes  of  interaction  with  HMDs  and  other 
devices  on  Pixel-Planes  4. 

•  The  pre-prototype  “bathysphere”  system  is  working.  The  "bathysphere"  uses  head-tracked  stereo  to 
achieve  an  out-the-window  view  of  a  virtual  world. 

•  We  built  and  put  in  use  new  hand-held  input  devices. 

2.2  See-through  Head-Mounted  Display  with  Custom  Optics 

•  We  have  completed  the  optical  design  of  a  3(Wegrec  field-of-view  system  using  off-the-shelf  lenses. 

•  We  assembled  and  tested  the  30-degree  field-of-view  system  on  an  optical  bench  in  our  optical  lab. 

•  The  diverse  t^tical  elements  are  currently  in  the  mechanical  shop  being  assembled  into  a  HMD. 

•  We  completed  the  optical  design  of  a  60-degree  field-of-view  system  using  custom  optics. 


2.3  Tracking 

*  We  c(xistnx;ted  a  full-scale  q>tical  tracking  system  in  time  to  be  demonstrated  to  the  public  at  the  ACM 
SIGGRAPH  '91  conference  (28  July-2  August  1991).  The  system  features  a  scalable  work  area  that 
cunently  measures  10'  x  12',  a  measurement  update  rate  of  20-100  Hz  with  20-60  ms  of  delay,  and  a 
resolution  specification  of  2  mm  and  0.2  degree.  The  sensors  consist  of  four  head-mounted  imaging 
devices  that  view  infrared  light-emitting  di^es  (LEDs)  mounted  in  a  10'  x  12'  grid  of  modular  2'  x  2' 
suspended  ceiling  panels.  Photogrammetric  techniques  allow  the  head's  location  to  be  expressed  as  a 
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function  the  Iax>wn  LED  positions  and  their  ejected  images  on  the  sensors.  The  woik  area  is  scaled 
by  simply  adding  panels  to  the  ceiling’s  grid,  rhsctxtdnuities  that  occurred  when  changing  working  sets 
of  LEDs  were  reduced  by  carefully  mana^g  all  error  sources,  including  LED  placement  tolerances,  and 
by  adopting  an  overdetermined  mathematical  model  for  the  computation  of  head  position:  space 
resection  collinearity.  To  our  knowledge,  this  is  the  first  demonstrated  scalable  tracking  system  for 
HMDs. 

•  After  its  return  fiom  SIGGRAPH  '91,  the  optical  tracker  was  installed  in  our  graphics  laboratory,  where 
it  runs  today.  Further  improvements  to  the  system  since  then  include; 

•  Doing  hand-tracking  by  installing  a  Polhemus  source  on  the  head  unit  and  a  sensor  on  the  hand.  The 
Hmittd  range  of  the  Pc^emus  is  not  a  problem  here  because  it  is  physically  iinpossible  to  move  your 
hand  out  of  range  and  magnetic  trackers  are  not  subject  to  line-of-si^t  restrictions  that  make  optical 
hand  tracking  a  difficult  problem.  This  deitx>nstrates  a tracker  system,  using  difierent 
technologies  to  cover  each  other's  weaknesses.  The  hand  controller,  falcated  in-lwuse,  is  a 
modified  bike  glove  with  two  buttons  for  input 

•  Inqnoving  the  heuristics  for  choosing  which  LEDs  to  light  increasing  the  average  update  rate  and 
reducing  the  average  latency. 

.  Finding  and  acquiring  a  set  of  smaller,  lighter  lenses.  We  will  attempt  recalibration  with  these  new 
lenses.  If  they  work,  they  will  remove  more  than  2  lbs.  of  weight  from  the  head. 

-  Recording  capabilities  have  been  added  (timing  and  position),  which  helps  us  collect  data  for  future 
tracking  studies. 

•  The  simulator  for  this  tracker  has  been  updated  to  reflect  its  current  configuraticMi  and  capabilities. 

•  We  wrote  a  conference  paper  and  a  technical  report  describing  the  optical  tracker  (see  Appendix  D). 

•  We  have  begun  informal,  interview-based  user  studies  of  the  intact  of  the  q>tical  tracker  on  the  design 
of  a  kitchen  .  This  is  being  done  in  conjunction  with  the  UNC  Walkthrough  group. 

•  In  conjunction  with  Dr.  John  F.  Hughes  of  Brown  University,  we  are  developing  calibration  techniques 
for  reducing  systematic  error  sources  in  the  optical  tracker. 

•  We  have  begun  studies  exploring  the  feasibility  of  predictive  tracking  (library  research,  initial 

experiments  with  Kalman  filters,  data  collection  our  qrtical  tracks). 

•  Efforts  to  build  a  head-tracked  stereo  display  system  are  currently  focusing  on  overcraiing  problems 
with  the  stereoscopnc  display.  In  the  interim,  tracking  is  being  done  by  a  magnetic-based  syscm  (the 
Bird),  but  we  reserve  die  option  to  replace  that  with  an  inside-looking  optical  system,  using  hardware 
similar  to  that  used  in  the  existing  optical  tracker. 

•  We  fabricated  prototyp)e  Self-Tracker  sensors  through  MOSIS. 

•  We  completed  the  interface  control  board  for  the  cluster  of  Self-Tracker  chips. 

•  The  design  for  the  software  simulation  of  Self-Tracker  chips  using  Pixel-Planes  5  is  in  progress. 
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2.4  Interactive  Building  Walkthrough 

•  The  Walkthrough  system  (both  the  tieadmiU  and  ceiling  tracker  versions)  was  successfully  demonstrated 
in  the  Tomorrow’s  Realities  Gallery  at  the  ACM  SIGGRAPH  ‘91  conference  in  Las  Vegas. 

•  We  have  added  a  capability  to  display  models  in  real-time  on  Pixel-Planes  5  as  they  are  being  built  and 
modified  on  Ae  Macintosh  using  the  Virtus  WalkThrtxr^  program. 

•  We  developed  textures  to  increase  the  model  realism,  including  a  flickering  fire  texture  [Rhoades92], 
with  appropriate  lighting  effects. 

•  We  added  location-dependent,  mcmaural  sound  cues  to  the  system. 

•  We  have  begmi  numing  user  studies  with  Walkduough  under  the  ceiling  tracker  to  study  the 
effectiveness  and  shortcomings  of  the  existing  system. 

•  We  have  installed  in  pilot  versions  of  our  code  the  ability  to  "q)en"  doors  in  the  model. 

3.0  Expected  Milestones  during  the  Next  12  Months 

3.1  Head-Mounted  Display  System 

•  Integrate  Vlib  (which  is  polygonally  based)  with  the  volume  rendering  work  on  Pixel-Planes  5. 

•  Integrate  ultrasound  input  more  uniformly  into  a  virtual  world. 

•  Build  and  integrate  an  interim  30-degree,  see-through  HMD. 

•  Conduct  basic  perceptual  experiments  on  what  sorts  of  virtual  objects  can  be  successfully  superimposed 
on  the  real  woiid  with  a  see-through  HMD. 

•  Experiment  with  new  types  of  hand-held  input  devices. 

•  Have  the  multiscreen  bathysphere  working  and  integrated  with  vlib. 

•  Perform  a  user  study  to  explore  the  advantages  of  head-tracked  steering  in  targeting  of  radiation  therapy 
treatment  beams. 

•  Calculate  tracker  delay  compensation  using  predictive  algorithms  for  the  HMD  system 

•  Be  able  to  explore  and  modify  molecular  surfaces  in  real  time  with  force  feedback  and  an  HMD. 

3.2  Optics 

•  Set  up  another  bench  prototype  of  the  30-degree  field-of-view  system  using  the  same  optics  and  robust 
and  adjustable  mechanical  smictures  to  cany  out  perceptual  stuches. 

•  Build  the  first  working  prototype  60-degree,  see-through  HMD  system. 

-  Mock-up  the  effect  of  a  10mm  pupil  for  a  60-degree  field-of-view  optical  system  in  the  optical  lab. 

-  Assemble  and  test  the  60-degree  field-of-view  system  on  the  optical  bench. 

-  Head  mount  the  60-degrce  field-of-view  system 
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Tracking 

•  C(»)tinue  inaemental  inqiiovements  in  Ac  optical  tracker.  Desired  goals  include: 

-  Reducing  the  weight  of  the  head  unit,  first  widi  light  lenses  and  later  by  designing  a  new  head  unit 
with  integral  photodiodes  and  lenses  to  reduce  the  weight  further  and  provide  greater  range  of  head 
motion. 

•  Increasing  the  ceiling  structure  to  around  20'  by  20'  in  order  to  provide  much  greater  range  of  motion, 
botii  quantitatively  and  psychologically. 

•  Investigating  the  feasilnlity  of  making  the  system  wireless.  This  tracker  actually  encourages  users  to 
walk  around  a  large  area,  running  the  risk  of  tripping  over  the  supporting  cables. 

•  Optimizing  low-level  code  to  iianease  performance. 


•  Continued  exploration  of  calibration  techniques  that  will  make  cellular  tracking  systems  easier  to 
construct  and  have  better  performance. 

•  Develop  and  demonstrate  predictive  tracking  techniques  for  HMDs.  Learn  what  is  required  in  the  system 
to  support  such  techniques  and  discover  what  limitations  they  have. 

•  Exploring  technologies  for  unlimited  range  tracking  in  unstructured  environments.  We  know  of  two 
technologies  that  have  this  potential:  Self-Tracker,  which  we  are  developing,  and  inertial  technologies. 
Both  are  relative-mode  trackers  that  measure  only  the  relative  differences  in  position  and  (mentation  as 
the  user  moves,  integrating  these  differences  over  time  to  recover  the  head's  Icx^tion.  Ibe  main  problem 
with  these  technologies  is  drift,  since  repeated  integration  accumulates  error  over  time.  Initial  systems 
will  probably  be  hybrids  with  our  optic^  tracker,  using  it  to  provide  auxiliary  information  to  evaluate  the 
technology  and  help  it  overcome  dr^  by  occasionally  proving  it  with  a  go^  fix  of  the  true  location. 

•  Continue  Self-Tracker  develc^ment. 

-  Develop  Self-Tracker  report  processing  algorithms. 

•  Complete  full  software  simulation  of  Self-Tracker  chips. 

-  Integrate  Self-Tracker  chips  with  board-level  interface. 


3.4  Interactive  Building  Walkthrough 

•  Come  up  with  an  interactive  model  desigii,  with  one  participant  using  the  UNC  Ceiling  Tracker  to  make 
design  decisicms,  and  another  using  the  Virtus  WalkThrou^  mcxiellLig  program  to  carry  them  out. 

•  Add  real-time,  interactive  radiosity  to  the  display  program  cm  Pixel-Planes  S. 

•  Add  real-time  model  mesh-generation  to  the  display  program.  This  is  under  development  by  a  visiting 
student  as  pan  of  his  master’s  work. 

•  Condua  additional  user  studies. 
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4.0  Discussion  of  Research 


4.1  Head’Mounted  Display  System 
Hardware 

The  current  HMD  system  is  used  by  many  projects  and  subprojects  in  a  variety  of  configurations.  The 
following  is  an  overview  of  the  cunent  system: 


Figure  1.  Overview  of  UNC  Head-Mounted  Display  System 


A  description  of  each  component  follows: 

•  Host  computer  A  Sun  4,  which  runs  the  host  side  of  the  application  and  interfaces  to  the  graphics 
engine. 

•  Graphics  Engine:  Pixel-Planes  5,  a  message-passing  multicomputer  capable  of  drawing  more  than  two 
millitm  Phong-shaded.  Z-buffered  trian^es  pCT  second.  Multiple  applications  can  run  on  the  system  at 
the  same  time  due  to  its  ability  to  be  split  into  multiple  subsystems. 

•  Magnetic  trackers:  Currently  SSPACEs  and  Birds,  but  we  will  be  beta  testing  new  nxxlels  soon,  and 
hope  to  increase  our  system’s  performance  with  fasrer,  newer  models  from  Polhemus  and/or 
Ascension. 

«  Ceiling  tracker.  Described  in  Section  4.3. 

•  Force  feedback  ARM:  Simulates  forces  in  the  virtual  world;  has  six  degrees  of  freedom  (3  forces  and 
3  torques);  uses  computer-controlled  servo  motors. 
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•  Manual  input  devices:  Cunently  bollowed-out  billiard  balls  with  Polhemus  sensors  mounted  inside, 
and  switches  on  the  outside  for  signaling  user  acdons.  Also,  a  Polhemus  sensor  mounted  on  a  guitar 
finger  pick  equipped  with  a  small  switch,  which  is  well-suit^  for  simple  manipulation  tasks.  We 
should  soon  have  a  new,  four-button  input  device  for  experimentation. 

•  Sound:  CcmtioUed  by  Macintosh;  plays  sounds  under  application  control  either  in  earphones  attached 
to  HMD,  or  fiom  sepiuate  speakers. 

In  general,  an  application  can  run  witii  almost  any  subset  of  the  above  equipment. 


Software 


Figure  2.  Overview  of  UNC  Virtual-Worlds  Software. 

(Dashed  lines  indicate  application  calls;  solid  lines  indicate  inter-library  calls.) 


The  virtual-worlds  base  software  is  fairly  complete  but  will  continue  to  expand  to  be  more  flexible  and 
powerful  and  to  accommodate  new  hardware.  Most  of  the  libraries  suppon  multiple  technologies 
transparently,  allowing  application  code  to  switch  easily  between  different  pieces  of  hardware  by  changing 
only  an  envircmment  vari^le.  While  efforts  at  improving  these  libraries  wUl  continue,  we  will  work  to 
add  new  capabilities  and  develop  new  applications.  The  software  goals  are  listed  in  with  the  other  twelve- 
month  goals  for  the  project  in  Section  3.1. 


4.2  Optics 

We  successfully  completed  the  design  of  a  30-degree  field-of-vicw  (FOV)  optical  system  using  off-the- 
shelf  optical  elements.  The  system  consists,  for  each  eye,  of  a  semi-transparent  mirr  jr  folded  at  45 
degrees  and  of  two  off-the-shelf  lenses.  In  the  head-mounted  version  of  this  system,  we  will  use  a 
240x360  color  LCD  display  with  a  display  area  of  54  x  40  mm.  The  effective  resolution,  however,  is 
240x120  because  of  the  triad  pixels  us^  for  color.  A  ray  tracing  of  the  system  is  shown  in  Figure  4.1. 
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This  system  was  assembled  and  tested  on  the  opdcal  bench  using  a  pair  of  stereo  slides  as  the  display,  and 
it  is  now  being  assembled  in  a  helmet  configur^on.  The  main  ^vantage  in  building  this  prototype, 
despite  a  FOV  limited  to  30  degrees,  is  to  test  some  of  our  ideas  in  terms  of  helmet  (ksigns  with  optics 
integration.  This  prototype  will  be  li^tweight  and  of  low  cost  It  is  also  expected  to  be  functional  within 
a  few  months,  which  w^  allow  us  to  study  more  th^ughly  perception  issues  related  to  see-through 
HMDs.  We  also  plan  to  set  up  another  version  oi  this  prototy^  on  the  t^tical  bench  in  order  to  carry  out 
specific  perceptual  studies  that  do  not  require  walking  around  in  space,  such  as  p^eived  depth  and  size  in 
such  a  "semi-virtual"  world.  Having  a  bench  prototype  can  also  allow  more  flexible  experiments  and 
adjustments  that  may  not  be  available  in  the  heaf^L 

The  optics  for  a  60  degree  field-of-view  HMD  with  a  dlted  catadiqrtric  combiner  have  been  designed,  and 
we  are  pr^aring  to  begin  building  it  in  a  few  mondts.  The  basic  d^gn  is  coinposed  of  a  tilted 
catadioptric  coi^riner,  a  relay  lens,  and  a  one-inch  CRT  color  display.  An  optical  layout  of  the  lens  is 
shown  in  Figure  42.  The  most  prominent  problems  or  difficulties  with  most  wide-angle  HMDs  are: 
qnall  eye  reOef;  small  pupil  sizes;  vignetting  at  the  edge  of  the  FOV;  the  increase  in  weight  due  to  the 
necessity  for  lar^  optical  coi^nents,  as  well  as  the  loss  of  image  quality,  as  the  pupil  size  increases;  the 
loss  in  transmission  at  the  c^tical  combiner  level;  and  spurious  reflections  fiom  unwanted  sources  in  the 
environmenL  This  system  will  allow  for  the  wearing  of  noost  eye  glasses,  and  the  pi^il  size  will  be 
sufficient  to  allow  rotation  of  the  eye  in  a  cone  of  60  degrees  without  causing  vignetting  of  some  pan  of 
the  FOV.  One  of  the  main  advantages  of  our  system  design  is  expected  to  be  freedom  from  spurious 
reflections. 

The  novelty  in  our  approach  to  a  wide-angle,  see-through  HMD  is  the  use  of  a  catadioptric  combiner  tilted 
at  about  30  degrees  from  the  optical  axis  of  the  eye  when  looking  straight  ahead.  Such  a  tilt  of  the 
combiner  can  lead  to  complex  cqrtical  systems  for  the  relay  lens  ^at  interface  the  display  and  the  combiner. 
Moreover,  the  optical  performance  of  a  tilted  system  becomes  asymcKtric  with  the  tilt  angle,  and  additional 
tilred  and  decentered  components  are  often  required  to  achieve  good  image  quality.  We  have  succeeded  in 
the  design  of  an  optical  system  that  is  composed  of  a  single  optical  element  comb^er,  five  lenses  for  the 
relay  lens,  only  two  difrerent  optical  glasses,  and  small  tilts  and  decenters  for  the  relay  lens  elements.  The 
system  is  designed  to  have  a  direct  transmission  of  67%  from  the  display  to  the  eye,  30%  see-through 
transmission,  and  is  to  be  free  of  spurious,  unwanted  reflections.  The  ratio  of  refiection/transmission  of 
the  tilted  combiner  can  be  chosen  acccxding  to  the  needs  of  the  application.  A  higher  direct  transmission 
than  the  see-through  transmission  may  be  desirable  in  our  applications  since  the  brighmess  of  the  CRT  is 
often  limited.  Figure  4.2  shows  that  ^e  relay  lens  is  compos«l  of  a  prism-like  element  forcing  the  ray 
bundle  to  fold  around  the  head,  a  triplet  lens,  and  a  field  lens.  The  system  is  color  corrected,  through  the 
use  of  two  different  qptical  glasses. 

The  lens  shown  in  Hgure  4.2  has  been  optimized  for  a  pupil  size  of  10  mm  and  an  effective  eye  relief  of 
23  mm,  as  specified  on  the  figure.  The  performance  for  an  unvignened  10  mm  pupil  is  represented  by  the 
tangential  and  sagittal  ray  fan  plots  for  1 1  representative  field  an^es  in  Figure  4.3:  on  axis;  vertical  fields 
of  ±15  degrees,  ±30  degrees;  horizontal  fields  of  15  degrees,  30  degrees;  and,  finally,  diagonal  fields  of 
±15  degrees  and  ±30  degrees.  The  effective  pupil  size  at  a  given  moment,  however,  is  dictated  by  the 
pupil  size  of  the  eye,  that  is  ^  mm  in  diameter  for  photopic  vision.  The  ray  fans  are  shown  in  Figure  4.4 
for  an  effective  pupil  size  of  3mm  One  of  the  optical  aberrations  that  increases  quadratically  with  the  FOV 
and  linearly  with  the  pupil  size  is  astigmatic  blur.  A  plot  of  astigmatic  blur  for  a  3  mm  pupU  is  shown  in 
Figure  4.5  across  a  60x60  degree  FOV.  The  effective  area  used  by  the  CRT  is  drawn  as  a  circle 
superimposed  on  the  field  lines.  This  plot  reveals  that  astigmatic  blur  is  negligible  at  everything  except 
deg^s  diagonal  FOV.  A  further  investigaticm  of  whether  or  not  this  astigmatic  blur  needs  to  be 
reduced  will  be  carried  out  when  looking  at  the  modulation  transfer  function  (MTF)  plots,  which  are 
widely  accepted  as  the  standard  ass<*ssment  of  image  quality. 

Another  optical  aberration  that  does  not  affect  image  quality  but  does  affect  the  geometry  of  the  image  is 
optical  distortion.  We  already  know  that  such  an  off-axis  design  will  not  be  fru  from  ^stortion.  Tbis  is  a 
trWle-off  between  having  an  on-axis  design,  low  distortion  system,  but  spurious  reflections  among  other 
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inconvenient  features,  and  an  off-axis  system  widi  higfi  distortion  but  free  from  unwanted  on-axis 
feamres.  Moreover,  (^mcal  distortion  can  be  compensated  for  electronically  by  predistorting  the  QIT 
image  using  a  lookup  table.  This  allows  more  degrees  of  freedom  for  the  system  which  can  then  be 
opdaized  to  reach  very  high  resolutions.  A  plot  of  the  optical  distcxiion  is  shown  in  Figure  4.6.  The  plot 
shows  a  mixture  of  keystone  and  banel  distohion,  as  expected.  We  plan  to  correct  for  the  distortion 
electronically,  and  some  mockups  of  it  are  currently  unclCT  development  with  members  of  the  Pixel-Planes 
tMm  We  thought  one  possible  tradeoff  might  be  the  speed  of  the  image  update,  but  a  closer  look  at  the 
problem  seems  to  indicate  that  the  Pixel-Planes  architecture  design  is  such  diat,  even  with  distortion 
correction,  the  update  of  images  should  not  be  affected.  We  woidd  ccxisider  a  hardware  implementation  of 
^e  distortion  compensation  if  image  generation  speed  were  to  become  a  problem. 

The  fiiuil  analysis  of  image  quality  is  shown  by  the  diffraction  limited  MTF  plots  shown  in  Figure  4.7  for 
a  pupil  of  3  mm.  A  plot  for  each  of  the  eleven  representative  field  angles  described  above  is  given  for  the 
reference  wavelengtii  of  559  nm  here  since  the  effect  of  residual  chromatic  abenaticxis  can  be  seen  on  the 
ray  fan  plots.  We  can  see  that  die  performance  is  exceptional  for  most  of  the  field  points  and  still 
acceptable  at  the  worst  astigmatic  point  in  the  field,  hideed,  a  high  resolution  of  1024x1024  corresponds 
to  a  cutoff  spatial  frequency  of  20  cycles/mm  while  we  show  the  performance  up  to  40  cycles/rom  to 
demcmstrate  the  high  potential  in  resolution  of  this  system.  We  also  looked  at  the  polychromatic  MTF 
plots  which  can  give  some  insight  into  the  strength  of  residual  chromatic  aberrations,  and  the  performance 
holds  very  close  indeed  to  the  monochromatic  cwie.  It  is  better  to  consider  the  monochromatic  plots, 
however,  in  terms  of  judging  the  image  quality  of  the  designed  system. 
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Figure  4.1.  Optical  Layout  of  the  30-de^e  FOV  see-through  HMD.  The  system 
is  composed  of  an  LCD  display,  an  inaging  lens  made  of  two  lenses,  and  a  folding 
mirror.  The  effective  eyerelief  (ER),  as  specified  in  the  figure,  is  17mna,  which 
should  allow  for  die  wearing  of  small  eyeglasses. 


Figure  4.2.  Optical  Layout  of  the  60-degree  FOV  see-through  HMD.  The  system 
can  be  thought  of  as  composed  of  a  tilted  combiner,  a  relay  lens  made  of  five 
elements,  and  a  one-inch  CRT  display.  The  effective  eyerelief  in  this  case  is 
23mm,  which  should  allow  for  the  wearing  of  most  eyeglasses. 
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Figure  4.3.  Tangential  and  sagittal  ray  fan  plots  for  1 1  field  angles  for  a  pupil  size  of  10mm. 
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III 


Figure  4.4.  Tangential  and  sagittal  ray  fan  plots 
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Figure  4.5.  Astigmatic  line  image  over  a  60x60-dcgrec  FOV.  The  circle  drawn  on 
this  picture  represents  a  circular  FOV  of  60  degrees.  This  is  a  typical  image  of  an 
optical  system  well  compensated  for  astigmatism.  Note  the  scale  chosen  to 
represent  the  lines. 


HORIZONTAL  FOV 


Figure  4.6.  Plot  of  an  undistorted  image  on  the  CRT  when  projected  without  any 
electronic  distortion  compensation. 
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4.3  Tracking 

It  is  well  known  that  trackers  for  HMDs  require  high  resolution,  high  update  rate,  and  low  latency. 

Range,  however,  has  not  received  similar  emphasis.  Many  virtual  worlds,  such  as  architectural 
walkthroughs,  would  benefit  from  more  freedom  of  movement  than  is  provided  by  existing  trackers. 
Long-range  trackers  would  allow  greater  areas  to  be  explored  naturally,  on  foot,  r^ucing  the  need  to 
resort  to  techniques  such  as  flying  or  walking  on  treadmills.  Also,  such  techniques  of  extending  range 
work  only  with  closed-view  HMDs  that  completely  obscure  reality.  With  see-through  HMDs,  which  we 
are  attempting  to  build  for  medical  applicadons,  the  user's  visual  ccmnecdon  with  re^ty  is  intaa  and 
hybrid  applications  are  possible  where  physical  objects  and  computer-generated  images  coexist.  In  this 
situation,  flying  though  the  model  is  meaningless.  The  model  is  registered  to  the  physical  world  and  one's 
relationship  to  both  must  change  simultaneously. 

For  the  short-term,  we  have  pursued  an  optical  technology  that  will  let  us  track  a  HMD  inside  a  room-sized 
environment,  which  we  call  the  ceiling  tracker.  The  ceiling  tracker  gives  us  room-sized  tracking  capability 
today  and  w^  be  a  vehicle  for  tracker  and  HMD  application  research.  But  for  the  long  term,  we  will 
pursue  relative-mode  trackers,  such  as  Self-Tracker  or  inertial  systems,  that  allow  unlimited  range  tracking 
in  unmodified  environments.  Since  such  technologies  suffer  from  drift  problems,  the  ceiling  tracker  will 
be  used  as  a  testbed  and  an  aid  in  their  development. 

Ceiling  Tracker 

The  existing  full-scale  prototype  is  the  second  generation  of  an  optoelectronic  head-tracking  concept  that 
originated  at  UNC  Chapel  Hill.  In  the  first  incarnation,  Jih-Fang  Wang  demonstrated  the  validity  of  the 
approach  in  a  three-sensor,  three-1  .ED  benchtop  prototype  [Wang90a].  The  current  system  places  four 
outward-looking  image  sensors  on  the  wearer's  head  and  locates  LEDs  in  a  10'  x  12'  suspended  ceiling 
structure  of  modular  2'  x  2'  ceiling  panels.  Each  panel  houses  32  LEDs,  for  a  total  of  960  LEDs  in  the 
ceiling.  Images  of  LEDs  are  fom^  by  lateral-effect  photodiode  detectors  within  each  head-mounted 
sensor.  The  location  of  each  LED's  image  on  a  detector,  or  photocoordinate,  is  used  along  with  the 
known  LED  locations  in  the  ceiling  to  compute  the  head's  position  and  orientation.  To  enhance  resolution, 
the  field  of  view  of  each  sensor  is  narrow.  Thus,  each  sensor  sees  only  a  small  number  of  LEDs  at  any 
instant.  As  the  user  moves  about,  the  working  set  of  visible  LEDs  changes,  making  this  a  cellular  head¬ 
tracking  system. 

Measurements  of  head  position  and  orientation  are  produced  at  a  rate  of  20- 100  Hz  with  20-60  ms  of 
delay.  Typical  average  values  are  50-70Hz  and  30ins  of  lag,  provided  that  the  user  stands  erect  and  keeps 
three  or  more  sensors  aimed  at  the  ceiling.  The  system's  accuracy  has  not  been  measured  precisely,  but 
the  resolution  is  2  mm  and  0.2  degrees.  It  was  demonstrated  in  the  Tomorrow's  Realities  gallery  at  the 
ACM  SIGGRAPH  '91  conference,  and  is,  to  our  knowledge,  the  first  demonstrated  scalable  he^-tracking 
system  for  HMDs. 

The  system  is  novel  for  two  reasons.  First,  the  sensor  configuration  is  unique.  Other  optical  tracking 
systems  fix  the  sensors  in  the  environment  and  mount  the  L^s  on  the  moving  body.  'The  outward- 
looking  configuration  is  superior  for  it  improves  the  system’s  ability  to  detect  head  rotation.  The  scalable 
work  space  is  the  system’s  second  contribution.  If  a  larger  work  space  is  desired,  more  panels  can  be 
easily  added  to  the  overhead  grid. 

This  system  extends  Wang's  work  in  several  ways.  First,  an  overhead  grid  of  960  LEDs  was  produced 
with  well -con  trolled  LED  location  tolerances,  and  more  attention  was  paid  to  controlling  other  error 
sources  as  well.  Second,  mathematical  techniques  were  developed  that  allow  an  arbitral  number  of 
sensors  and  an  arbitrary  number  of  LEDs  in  the  field  of  view  of  each  sensor  to  be  used  in  the  computation 
of  head  location  [Azuma91].  This  resulted  in  an  overdetermined  system  of  equations  which,  when  solved, 
was  less  susceptible  to  system  error  sources  than  the  previous  mathematical  approach  (Church's  Method). 
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Third,  the  analog  signals  emerpng  from  the  sensors  were  digitally  processed  to  reject  ambient  light 
Finally,  techniques  for  quickly  detennining  the  working  sets  of  lM>s  were  developed. 

Experience  widi  simulations  and  an  eariy  48-LED  prototype  revealed  the  problem  of  beacon  switching 
error,  as  the  user  nxrved  around  and  the  working  set  of  beaccxis  changed,  discontinuous  jumps  in  position 
and  orientadcMi  occurred.  These  are  caused  by  errors  in  dre  sensor  locatitxis,  distortions  causal  by  the  lens 
and  photodiode  detector,  and  errors  in  the  positions  of  the  beacons  in  the  cdling.  We  controlled  these 
errors  by  measuring  the  posidons  of  the  sensors  on  the  head  unit  with  a  coordinate  measuring  machine, 
calibrating  the  lens-dete(^  assemblies,  and  housing  the  LEDs  inside  carehilly-constiucted  panels  arid 
nx>unting  those  in  a  ceiling  superstructure  that  was  Celled  with  a  Spectra  Physics  Laser-L^el. 

The  mathenutical  routine  used  to  recover  die  posidon  and  orientadon  of  the  user's  head  is  a  variadon  of  a 
photogrammetric  techi^ue  called  space-resecdon  by  collinearity  [Wolf83].  We  have  exrended  it  to  handle 
informadon  from  muldple  distribute  sensors.  At  least  three  LEDs  nwd  to  be  seen  to  compute  a  soludon. 
If  more  are  seen,  this  routine  will  generate  the  best  soludon  in  a  least-squares  sense.  Using  more  than 
three  beacems  reduces  die  effects  of  errors  firom  systemadc  error  sources.  The  Collinearity  module  runs  on 
an  i860  board  and  takes  about  6  ms  to  run  under  normal  circumstances. 

Many  steps  were  taken  in  the  low-level  software  and  hardware  to  maintain  high  signal-to-noise  rados  and 
achieve  good  resoludon  from  our  sensors.  Ambient  light  was  rejected  by  placing  filters  on  the  lenses  and 
by  taking  readings  with  the  LED  off  and  subtracting  ttose  values  from  the  signals  with  the  T  (xi.  When 
sensing  an  LED,  many  samples  are  taken  and  averaged  to  reduce  random  noise  effects.  The  distance 
between  a  sensor  and  an  T.Fn  varies  widely  because  the  user  is  moving  around,  so  the  current  used  to  light 
an  LED  is  adjusted  dynamically  to  maximize  signal  without  saturating  the  detectors. 

At  any  given  instant,  the  number  of  visible  LEDs  is  small  compared  to  the  total  number  of  LEDs  in  the 
ceiling,  so  discovering  what  this  visible  set  is  requires  somethmg  faster  than  a  brute-force  scan  of  the 
entire  ceiling  array.  Assumptions  of  spatial  and  terr^xxal  cc^erence  let  us  develop  more  efficient 
heuristics:  1)  Given  the  last  known  set  of  beacons,  test  those  and  their  neighbors,  if  needed.  2)  Given  that 
we  are  confuting  the  posidon  and  orientadtm  of  die  head,  use  that  to  predict  which  LEDs  the  sensors 
should  see  if  the  head  is  actually  at  that  location. 

The  main  problem  with  the  existing  system  is  the  axnbinadon  of  excessive  head-bom  weight  and  limited 
rotation  range.  Rotation  range  depends  heavily  on  the  user’s  height  and  position  under  the  ceiling.  A 
typical  maximum  pitch  range  near  the  center  of  the  ceiling  is  45  degrees  forward  and  45  degrees  t^k. 
Given  the  current  focal  lengths,  simulations  show  that  as  many  as  eight  fields  of  view  are  required  for  a 
respectable  rotation  range  [Wang90b].  The  existing  head  unit  weighs  about  10  lbs.  (4  lbs.  for  the 
displays,  6  lbs.  for  the  head  frame  and  sensors).  The  weight  of  each  sensor  must  be  significantly  reduced 
to  achieve  this  goal.  To  reduce  weight,  we  are  trying  to  replace  the  current  lenses  (1 1  oz.  each)  with 
smaller,  lifter  lenses  (2  oz.  each).  Wang  also  proposed  optically  multiplexing  multiple  fields  of  view 
onto  on  a  single  lateral-effect  photodiode.  However,  it  may  be  easier  to  design  a  new  bead  unit  with 
integral  phot^odes  aixl  lenses.  Given  tiiat  each  photodiode  is  about  the  size  of  a  quarter,  the  entire 
surface  of  a  head  unit  could  be  studded  with  sensors. 
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Photos  of  the  optical  ceiling  tracker 


The  existing  system  in  UNC’s 
graphics  laboratory 


Conceptual  drawing  showing  fields  of 
view  of  the  head-mounted  sensors 


Head  unit  with  four  sensors  and 
supp>crting  electronics 


Sensors  viewing  LEDs  in 
the  ceiling.  Photo  taken  with 
a  camera  that  is  sensitive  to 
infrared  light. 
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Self-Tracker 

The  self-cracker  [Bishop84]  is  one  possible  inside-out  tracking  system  which  may  replace,  or  augment,  the 
optoelectronic  system.  Its  goal  is  to  eventually  eliminate  fixed  beacons  and  perform  tracking  using 
features  found  naturally  in  the  environment  The  self-tracker  could  then  be  taken  to  any  room,  hallway,  or 
even  outdoors. 

The  self-tracker  will  use  a  cluster  of  custom  made  VLSI  chips  which  will  be  mounted  on  the  head-mounted 
display  and  look  out  on  the  environment  in  multiple  directions.  The  cluster  of  chips  will  report  the 
^jparent  shift  of  the  image  seen  by  each  chip  during  successive  clock  cycles.  A  host  ctxnputer  will  collect 
the  information  and  calc^te  the  distance  the  user  ^  travelled  relative  to  his  previous  position. 
Approximately  1000  frames  per  second  will  be  required  to  track  the  fastest  pobible  movements  reported  in 
human  motion  literature.  Hus  system  may  not  w(rt  by  itself  because  of  accumulated  drift  Fortheinidal 
versions,  a  small  number  of  beacons  may  be  placed  in  the  environment  to  allow  for  error  correction. 

Each  self-tracker  chip  will  contain  a  one-dimensional  array  of  photosensors  and  supporting  image 
processing  circuitry.  A  one-dimensional  airay  will  be  used  so  that  a  sufficiendy  lai^e  number  of  pixels 
can  be  pl^ed  on  the  chip  and  so  that  image  correlation  can  be  performed  in  real  time.  The  correladon 
between  successive  frames  will  be  accomplished  by  converting  the  image  to  a  digital  representadon  and 
then  successively  shifting  the  posidon  of  one  frame  and  reporting  the  shift  which  corresponds  to  the 
closest  match  to  the  other  frame. 

The  tracker  chips  will  be  mounted  in  pairs  so  that  stereopsis  can  be  used  to  measure  the  distance  to  the 
observed  surface  in  the  scene.  Each  tracker  chip  will  report  the  observed  shift  in  terms  of  the  number  of 
pixels  of  shift,  so  the  distance  value  will  be  required  to  convert  the  shift  from  pixels  to  a  physical 
measurement  One  chip  in  each  pair  will  transmit  the  its  image's  distal  representadon  to  the  other  chip. 
The  second  chip  will  then  calculate  the  correladon  between  the  two  images  in  the  same  manner  that  it 
calculated  the  correladon  between  subsequent  images  the  same  chip. 

The  diagram  below  is  a  block  diagram  of  a  self-tracker  chip.  A  one-dimensional  photosensor  array 
generates  an  array  of  analog  voltages  which  represent  the  incident  light  on  the  array.  The  edge  dctecdon 
block  produces  a  digital  representadon  of  the  image.  Then,  the  image  comparison  block  compares  the 
current  image  with  ^e  image  seen  during  the  last  time  frame.  Hie  shift  between  the  two  images  is 
reported,  and  the  current  frame  is  stored  for  comparison  with  the  next  frame.  Also,  each  image  is  sent  to 
its  stereo  partner  to  calculate  the  distance  to  the  observed  image. 
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Environment 


I  I  i  I  I  |>-lght 


Analog 

Signal 


Digital 

Signal 


Image  Comparison 


i 

Image  Shift 
(sent  to  computer) 


Figjie  3.  Architecture  of  the  Self-Tracker  custom  VLSI  sensor. 

Over  the  past  six  months,  research  has  progressed  in  both  hardware  and  software.  A  set  of  prototype 
Self-Tracker  chips  with  a  complete  set  of  image  detection  and  processing  circuitry  has  been  fabricate 
through  MOSIS.  In  addition,  supporting  hardware  which  connects  the  Self-Tracker  chips  with  the  host 
computer  has  been  construaed.  A  software  model  of  the  Self-Tracker  chips  using  images  generated  by 
Ihxei-Planes  5  is  under  development  The  software  model  has  been  added  to  software  which  uses  images 
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captured  from  a  TV  camera.  The  model  has  been  used  to  refine  image  processing  circuitry  on  the  Self* 
Tracker  chips  to  reduce  the  level  of  errors. 

Inertial  Technologies 

Inertial  guidance  systems,  used  to  steer  planes,  ships,  guided  missiles,  etc.,  do  not  require  modification  of 
the  environment  and  have  no  range  restrictions.  They  could  be  nearly  ideal  for  tracking  HMDs  in  the 
future,  if  the  endemic  drift  problems  can  be  controlled.  Small,  light  accelerometers  are  available  today; 
small,  light  laser-ring  or  fi^-opdc  gyroscopes  are  just  on  the  threshold  of  availability  to  the  public  and  are 
very  expensive.  A  hybrid  approach  combining  inertial  technologies  with  a  system  that  occasionally 
pro^^  absolute  locadoo  fixes  to  combat  drift  could  be  highly  effecdve. 


4.4  Interactive  Building  Walkthrough 

The  UNC  Interacdve  Building  Walkthrou^  Subproject  aims  at  the  development  of  systems  to  help 
architects  and  their  clients  explore  a  building  design  prior  to  its  construcdon  and,  therefore,  correcdng 
problems  on  the  computer  instead  of  in  concrete  $rooks86].  The  scenes  are  shown  with  radiosity 
illuminadon,  lights  that  can  be  switched  on  or  off  with  only  100  msec  delays,  textured  surfaces,  and  near- 
real-dme  display  from  changing  viewpoints.  Update  rates  realized  by  the  I^el-Planes  5  graphics  scene 
generator  arc  30-40  updateVsecond  (15-20  stereo  images/second)  on  scenes  consisdng  of  thousands  of 
polygons,  many  of  which  are  textured. 

One  system  uses  a  head-mounted  display  and  the  new  UNC  Ceiling  Tracker  to  allow  the  viewer  literally  to 
walk  freely  about  a  10'  x  12'  living  room  in  an  imaginary  tqiartmenL  Both  systems  include  the  display  of 
scene-appropriate  sounds.  The  second  system  uses  our  UNC  steerable  treadmill  and  a  head-mounted 
display  to  allow  viewers  virtually  to  walk  around  a  fully  furnished  ten-room  house.  Different  appropriate 
sounds  are  heard  in  each  room  and  outdoors. 

We  have  long  held  that  virtual  worlds  projects  need  to  move  forward  simultaneously  in  four  dimensions: 
faster,  prettier,  handier,  and  realer.  In  the  past  year,  we  have  made  the  following  progress  in  each  of 
these  areas; 

•  Faster: 

The  Pixel-Planes  5  machine  came  on-line  last  year.  Over  the  course  of  the  year,  we  and  the  Pixel-Planes 
5  team  worked  together  to  remove  the  initial  hindrances  at  realizing  its  spc^.  We  transponed  Pixel- 
Planes  5  to  the  SIGGRAPH  '91  conference  and  demonstrated  its  ability  to  display  2  million  Phong- 
shaded,  on-screen  triangles/sccond;  this  is  about  twice  as  fast  as  the  state-of-thc-an  Silicon  Graphics 
machine.  Texturing  substantially  reduces  this  rate.  The  power  of  Pixel-Planes  5  is,  however,  sufficient 
for  dynamic  interaction  of  high-fidelity  images. 

•  Prettier: 

The  major  step  forward  this  year  has  been  the  addition  of  and-aliased  textures.  They  radically  improve 
the  visual  fidelity  of  a  scene  with  a  fixed  number  of  polygons.  We  have  added  textures  for  wood,  brick, 
ceiling  tile,  linoleum,  bathroom  tile,  etc.  We  even  use  textures  for  sheet  music,  piano  keys,  and  art 
hanging  on  the  wall.  Pixel-Planes  5  is  very  good  at  procedural  texmres;  it  is  not  especially  good  at  bit¬ 
mapped  (image)  textures. 

The  second  major  step  on  scene  fidelity  has  been  the  addition  of  location-dependent  (monaural)  sound 
cues.  We  adapted  commercial  Macintosh  software,  purchased  a  10,000-sound  CD  sound-cue  library, 
and  developed  the  location-based  sound  selection,  loudness  control,  and  sound-sound  mixing  software. 
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Handier: 


Wc  developed  new  user  interfaces  for 

•  Ae  ceiling  tracker 

•  interactive  turning  on  and  off  of  lights  in  the  model 

•  full  integration  of  our  treadmill  inurface  with  the  head-mounted  display  and  Polhemus  head 
tracker. 

•  Realer: 

We  gained  a  lot  of  experience  in  model  building  this  year.  The  house  model  was  refined,  and  a  sound 
Tpr>H<»i  was  built  We  also  built  de  novo  a  model  for  a  six-room  furnished  apartment.  On  the  theoretical 
side,  we  developed  a  whole,  new,  many-level  hierarchical  concept  of  how  architectural  models  should 
be  organized. 

In  the  past  only  the  display  stage  has  been  interactive,  and  that  interactivity  has  been  limited.  Although 
the  lights  could  be  controlled  interactively,  the  geometry  was  fixed.  Any  changes  in  the  tnodel  had  to  be 
sent  back  through  the  model  pipeline,  a  {vocess  that  could  take  days  or  weeks.  Changes  iri  the  model  were 
typically  limited  to  cOTrcctions  of  modelling  errors;  changes  in  design  were  prohibitively  time-consuming. 

Much  of  oui  present  activity  is  devoted  to  improving  the  system  interactivity,  to  decreasing  the  time 
between  model  design-cycle  iterations.  We  have  begun  a  collaboration  with  Virtus  Corporation,  using 
their  package,  Virtus  WalkThrough,  as  a  modelling  tool.  They  have  provided  a  version  of  Virtus 
WalkThrough  fiom  which  we  can  expem  a  model  to  Pixel-Planes  5  for  immediate  display.  One  of  our 
goals  is  to  be  able  to  do  interactive  design,  with  a  user  under  the  UNC  Ceiling  Tracker  niaking  design 
decisions  and  a  nxxicUer  using  Virtus  WalkThrough  to  carry  them  ouL  The  user  will  be  able  to  explore  a 
rrxxlel,  maitipulate  objects,  suggest  modifications,  and  have  those  modifications  implemented  in  the  nxxlel 
around  them. 
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5.2  Presentations 


By:  Frederick  P.  Brooks,  Jr. 

Tc^c:  Computer  Graphics:  More  Unsolved  Problems 

Event:  SIGGRAPH  '91  panel 

Place:  Las  Vegas,  NV 

Date:  July  28^August  2, 1991 

By:  James  Chung 

Topic:  Application  ^  Head-Mounted  Display  to  Radiotherapy  Treatment  Planning 

Event:  d^l  Docroral  Consortium 

Place:  New  Orleans,  LA 

Date:  April  27^ay  2, 1991 

By:  Henry  Fuchs 

Topic:  Scientific  Visualization  on  Advanced  Architectures 

Event:  SIGGRAPH  *91  panel 

Place:  Las  Vegas,  NV 

Date:  July  28-August  2, 1991 

By:  Richard  Holloway 

Topic:  Virtual-Worlds  Research  at  UNC-CH 

Event:  Computer  Graphics '91 

Place:  London,  England 

Date:  November  1991 

By:  Warren  Robinett 

Topic:  Head-Mounted  Display  Research  at  NASA  and  UNC-CH 

Event  Banff  Centre  for  the  Arts 

Place:  Banff,  Alberta,  Canada 

Date;  October  30,  IS^l 

By:  Warren  Robinett 

Topic;  Concrete  Portrayal  of  :.lathematical  Abstraction  Using  Computer  Graphics 

Event:  SIGGRAPH  '91 

Place:  Las  Vegas,  NV 

Date:  August  1991 

By;  Warrei:  Robinett 

Topic:  Expansion  of  Human  Perception 

Event  SRI  COTifertnce  on  Virtual  Reality 

Place:  Menlo  Park,  CA 

Date:  June  1991 

By:  Warren  Robinett 

Topic:  Virtual  Reality  for  Education 

Event  International  Conference  on  Education  and  Technology 

Place:  Toronto,  Canada 

May  1991 

By:  Jannick  Rolland 

Tqnc:  Insight  into  Optical  Properties  of  Wide-angle  See-through  Head-Mounted  Displays 

Event  SRI  Conference  on  Vir^  Reality 

Place:  Menlo  Park,  CA 

Date;  June  1991 
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5.3  Some  of  the  Visitors  who  observed  HMD  demos  at  UNC-Chapel  Hill 


•  Six  from  National  Center  for  Supercomputing  Applications 

•  Six  from  the  Math  and  Science  Center,  Richmond,  Virginia 

•  Daniel  Goldstein,  Swiss  jounudist 

•  Eugene  BylimJcy,  Fortune  Magazine 

•  Architecture  stui^nts  from  Guilford  College,  Greensboro,  NC 

•  Larry  Engel,  WGBH-TV,  Boston,  Mass. 

•  Qndy  Cc^lo,  UNC  Center  for  Math  and  Science  Education 

•  Rud)  Pachter,  Hm  Running,  and  Dave  Zeltnan,  Wright'Patterson  Air  Force  Base 

•  Gary  Diver,  Jim  Davis,  and  Bill  ToUhurst,  IV^  C^. 

•  Gottfried  Ungeiboeck,  Mark  Rinaldi,  Jim  Gray,  IBM 

•  Dr.  Karl  Zehnder,  ETH-ZOrich 

•  Ivan  Sutherland  and  Robert  SprouU,  Sun  Microsystems 

•  Charles  Grimsdale  and  Phil  Atkins,  Division  Ltd.,  Bristol,  UK 

•  Capt  Don  Stark  and  Gloria  Golden,  Fort  Huachuca,  AZ 

•  Vic  Reis  and  Jcdm  Toole,  DARPA/ISTO 

•  Attendees  of  the  DARPA  Prototyping  Spring  Contractors  Meeting  (30-35  people) 

•  Richard  Clinch,  North  Carolina  Board  of  Science  and  Technology 

•  Chief  Academic  Officers  frxxn  about  30  East  Coast  Universides 

•  Laurent  Belsie,  Christian  Science  Monitor 

•  Attendees  of  the  Intemadonal  Meeting  of  the  Molecular  Graphics  Society  (around  125  people) 

•  Deimis  Roark  and  Jim  Raumer,  SAIC 

•  30  Students  frmn  Program  for  Future  Math  and  Science  Teachers,  sponsored  by  NC  Governor's  Offi 

•  Randall  Shumaker  and  Dale  Ltxtg.  Naval  Research  Lab 

•  Bruce  I%illips,  North  Carolina  Museum  of  Life  and  Science 

•  John  Lang,  Hewlett  Packard 

•  Jay  Schafer,  Kim  Spencer,  and  crew  from  ABC-TVs  Prime  Time  Live 

•  John  Imholz,  Virginia  Tech  School  of  Architecture 

•  Toldichiro  Takahashi,  NTT,  Japan 

•  Phil  Ambum,  Air  Face  Institute  of  Technology 

•  Dave  Neyland,  DARPA 

•  Andy  G<^,  Hewlett  Packard 

•  Victor  Zadolotnyi  and  Michael  Hovan,  George  Masai  University 

•  Bill  Furness  and  Martin  Green,  1ST 

•  Steve  Ellis,  NASA  Ames  Research  Onter 

•  Takayuki  Nakajima,  Sharp  Corp.,  Tokyo 

•  Ben  Delaney,  Cyberedge  Joumd 

•  Len  Bass  and  Rick  Kazman,  Software  Engineering  Institute,  Carnegie  Mellon  University 

•  Sat  Narayanan  and  Richard  lOiapp,  Tektronix 

•  John  Hestenes,  National  Science  Foundation 

•  Dr.  Merten,  President,  Siemens  Research  Corp^tion 

•  Dr.  Ming-Yee  Chiu,  Head  of  Imaging  Dept,  Siemens  Research  C^ip. 

•  Lance  Glasser  and  Marko  Slusarezuk,  DARPA 

•  Ten  from  SAS  Institute,  Cary,  NC 

•  Rich  Gamer,  Bob  Liang,  and  Dan  Ling,  IBM  T.  J.  Watson  Research  Onter 

•  Capt  Brad  Hooker,  Fort  Huachuca,  ^ 

•  Eric  Melvin  and  Jane  Fanning,  Digital  Equipment  (!!oip. 

•  Wayne  Winstead  and  Jorge  Montoya,  Research  Triangle  Institute,  RTP,  NC 

•  Joe  La  Russa,  Electro  Visual  Engineering,  Yorktown  Heights,  NY 

•  Wim  van  Damme,  Univ.  of  Utrecht,  The  Netherlands 
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6.0  Appendixes 
Appendix  A 

Paper:  Butterworth,  Jeff,  Andrew  Davidson,  Stephen  Hench,  and  T.  Marc  Olano.  "SDM:  A  Three- 
Di^nsional  Modeler  Using  a  Head-Mounted  Display.”  To  appear  in  ACM  Computer  Graphics — 
Proceedings  of  the  1992  Symposium  on  Interactive  3D  Grapfiics  (Cambridge,  Mass.,  April  1992). 

Appendix  B 

Ptq)er  Chung,  James  C.  ”ACoiiq)arisonofHead-tracked  and  Ntm-head-tiacked  Steering  Modes  in  the 
Targeting  of  Radiother^y  Treatment  Beams."  To  appear  in  ACM  Computer  Graphics— Proceedings  of 
the  1992  Symposium  on  Interactive  3D  Graphics  (Cambridge,  Mass..  April  1992). 

Appendix  C 

Pap^  Robinett,  Warren,  and  Richard  Holloway.  Implementation  of  Flying,  Scaling,  and  Grabbing  in 
Virtual  Worlds."  To  appear  in  ACM  Computer  Grapt^s— Proceedings  of  the  1992  Symposium  on 
Interactive  3D  Graphics  (Cambridge,  Mass.,  April  1992). 

Appendix  D 

PapCTt  Ward,  Mark,  Ronald  Azuma,  Robert  Bennett,  Stefan  Gonschalk,  Henry  Fuchs.  "A  Demonstrated 
Optical  Tracker  with  Scalable  Work  Area  for  Head-Mounred  Display  Systems."  To  appear  in  ACM 
Computer  Graphics— Proceedings  of  the  1992  Symposium  on  Interactive  3D  Graphics  ((Cambridge, 
Mass.,  April  1992). 

Appendix  E 

Handouts  from  the  Head-Mounted  Display  demonstrations  in  the  Tomorrow's  Realities  Gallery  at  the 
SIGGRAPH  '91  conference  in  Las  Vegas,  NV,  (28  July  -  2  August  1991). 

•  Interactive  Building  Walkthrough  Using  the  Opdeal  Tracker 

•  3dm:  A  Two-Person  Modeling  System 

•  Flying  Through  Molecules 

•  Interactive  Building  Walkthrough  Using  A  Steerable  Treadmill 

•  Radiation  Thcr^y  Treatment  Planning 

•  Mountain  Bike 

•  Molecule  Museum 
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Abstract 

3dtn  is  a  three  dimensional  (30)  stirface  modeling 
program  that  draws  tachniques  of  mo^l  mantpulation  from 
both  CAO  and  drawing  programs  and  applies  them  to 
modeling  in  an  mtuiiive  way.  3dm  uses  a  head-mounted 
display  (HMD)  to  simplify  the  problem  of  3D  model 
manipulation  and  understandhig.  A  HMD  places  the  user  in 
the  modeling  space,  making  three  dimeiuional  relationships 
more  understandable.  As  a  result,  3dm  is  euy  to  learn  how  u 
use  and  encourages  expenmentadon  with  model  shapes. 

1  Introduction 

The  use  of  interactive  30  enviranmenu  has 
increased  the  demand  for  complex  3D  modcls.(9]  The  30 
environments  that  provide  a  sense  of  telepresence  or  -vinual 
reality-  require  a  large  number  of  models  in  order  to  give  the 
user  the  illusion  of  b^g  in  a  specific  place.  This  demand  for 
more  models  has  highlighted  the  fact  that  most  modeling 
systems  are  diflieult  to  use  for  all  but  a  small  number  of 
expertt.(9]  Through  identincation  and  removal  of  some  of 
the  fundamental  obstacles  to  modeling  ere  hope  to  make  it 
aceauibls  to  more  users. 

Typical  techniques  used  to  select  and  display 
objects  arc  a  major  hindrance  to  3D  modeling43]  To  placs  an 
object  in  3D  requires  six  paramsien:  the  position  (three)  and 
the  OTMtuatian  (three).  Most  modeling  systcais  (nwddcrs) 
must  settle  for  a  2D  mouse  augmented  by  a  keyboard  for  this 
purpose.  This  mismatch  resulu  in  difficult  placement  and 
.  picking  of  objectt  in  modeling  space.  The  dis^y  of  models 
usually  takes  the  form  of  a  projection  onto  a  2D  nwnitor. 
This  to  the  effect  of  making  spatial  rclaiioashipi  unclear. 
Technological  improvemenu  to  3D  model  display  and 
manipulation  hardware  can  remove  these  barriers  to  model 
creation  and  understanding. 

Currant  virtual  reality  technology  provides  one 
solution  to  more  inmitivs  modeling.  A  HMD  system  gives 
dM  ability  to  understand  complex  spatial  relationships  of 
.  models  by  placing  the  user  in  the  model's  world.  Within  this 
type  of  system,  a  hand-held  pointing  device  supplies  users 
with  the  ability  to  specify  3D  relationships  through  dirao 


3D  manipulation.  As  a  result,  the  user  can  build  the  virtual 
world  from  witAiit  the  virtual  world. 

Our  source  of  inspiration  for  designing  a  user 
interface  for  a  HMD-based  modeler  is  the  current  software 
used  for  2D  modeling.  At  one  time,  creating  2D  models 
required  cumbersome  CAD  programs.  This  software  took  a 
long  time  to  learn  and  often  did  not  provide  real-time 
interaction.  Now,  howeve,  20  drawings  can  be  manipulated 
by  even  the  most  casual  usen  of  personal  computers.  This 
revolution  is  in  part  the  result  of  intuitive  drawing  programs 
hke  MacDraw.  One  of  the  keys  to  MacDraw's  success  is  its 
inherent  simplicity.  Most  work  done  with  it  requires  no 
reading  or  use  of  the  keyboard.  Rather,  it  provides  a  palette 
of  tools  which  is  always  available  next  to  the  model.  To 
change  modes,  the  user  simply  selecu  the  tool  from  the 
palette  using  the  mouse.  The  process  of  3D  modeling  can 
become  more  accessible  if  some  of  the  lessons  learned  from 
this  .evaluation  of  2D  modeling  can  be  applied  to  3D 
modeling  systems. 

This  peprr  presenu  a  HMD-based  system  called 
3da  which  simplifies  the  task  of  3D  modeling  by 
implementing  the  concepu  introduced  above.  Basic 
tachniquas  for  working  within  3dra’s  virtual  worid  are 
daacrib^  id  show  how  users  access  the  various  features.  The 
impleroeatation  of  3dm  is  described  through  a  presenution 
of  its  moist  useful  commands.  Finally,  the  resulu  of  actually 
using  3dm  ve  presented  with  at  emphasis  on  new  techniques 
that  can  be  applied  within  ocher  virmal  worlds. 

2  Prior  Work 

A  lags  body  of  work  has  been  done  on  3D 
modeling.  Although  3D  I'lipur  devices  have  been  used  to 
enhance  modelers,  vary  little  modeling  has  been  done  with  a 
HMD.  Some  exampies  of  modeling  with  six  degree-of- 
freedom  input  devices  are  [1]  and  (8],  but  both  of  those  used 
traditional  2D  displays.  Pr^ious  uses  of  HMD  systems  have 
concentrated  more  on  exploration  of  virtual  worlds  rather 
than  creating  or  modifying  them.  Some  examples  of  this 
work  with  HMD's  can  be  found  in  {5]. 

Modeling  using  a  HMD  system  has  been  explored 
by  Oak.(4]  Users  of  Gark's  system  created  parametric 
surfaces  by  manipulating  control  points  on  a  wire-frame  grid 
This  system  highlighted  the  utility  of  using  a  HMD  for 
improved  understanding  and  interaction  with  models.  Like 
Clark's  system.  3dm  reiies  on  a  HMD  to  help  simplify 
modeling,  but  3dm's  intuitive  user  interface  design  also 
makes  it  easy  to  learn  and  use. 


f) 


3  Impicmvniulion 

.)(liii  was  tic%cUi|KMl  uoiii):  a  VI'l.  a>  the 

JiNpljy  device  and  Piilhciiiuv  iijckers  Ui  irjck  the  head  and 
hand.  A  hD  2’huium  iiumse,  deveK>|k'il  ai  I'NC'  CH.  »as  the 
input  device.  The  images  were  rendered  using  the 
Pixcl-l’Ianes  4  and  Pixel  Planes  high-(icrlormance  graphics 
engines  developed  at  UNC-CH.|h||7|  Cunently.  all  models 
created  with  3dm  are  made  up  of  hierarchical  groups  of 
triangles. 

3 . 1  User  Interface 

In  addition  to  the  model,  the  vinual  world  of  3dni 
contains  the  componems  of  the  user  interface.  The  most 
important  of  these  are  the  toolbox  and  the  cursor.  The  cursor 
follows  the  position  of  the  hand-held  mouse,  giving  the  user 
a  sense  of  hand  position  in  the  modeling  space.  The  toolbox 
is  the  means  by  which  most  actioiu  are  performed. 

Some  of  the  user  interface  components  me  simply 
helpful  markers  that  can  be  turned  off.  unlike  the  toolbox  and 
the  cursor,  which  are  always  visible.  The  user  stands  on  a 
“magic  carpel”  which  marks  the  boundaries  of  where  the 
tracking  system  operates.  Remaining  within  tracker  range  is 
important  because  the  virtual  world  will  begin  to  till  as  the 
user  moves  farther  out  of  range.  Below  the  magic  capei  lies  a 
checkered  ground  plane,  ebove  which  the  model  is  usually 
aeaied.  Additional  reference  objects,  such  as  coordinate 
axes,  can  be  aimed  on  by  the  user. 


|■|vr:llll>M  li'ggi.'  vlijiigi.'  Miiiu'  gliil'jl  u\|K'i  1  Ilf  tiliii  An 
cvjitipl.  1^  ilic  -II. i('  i*>  gfhl  iv»|;j;l,*.  wliitli  rc-iriciv  ,  iir-or 
iiiiivi'iiii’tii  ID  .1  <l>  gii.l  when  II  i\  Dll 

I  \pl.>riiie  III.'  iii.hI.-I  pi.ivi.le-  un.lersi.in.lmg  dI  ns 
.M)  shjpe.  Ml  .t.liii  MiptSDris  multiple  mcthmls  of  njvigjting 
in  ihe  iiiiHlcling  *|usc  Tlic  MMU  system  used  for  3dni  allows 
the  user  to  walk  through  the  mixlel  space  a  few  paces  in  any 
direction.  Walking  simply  dtKs  not  provide  ihc  range  of 
movcmcni  needed  for  most  models,  so  3dm  supports 
“flying."  a  commonly  used  method  of  traveling  through 
virtual  world.s.|2|  Flying  consists  of  translating  the  user 
through  model  space  in  the  direction  that  the  cursor  is 
pointing.  Rying  moves  the  magic  carpet,  which  carries  the 
user  and  the  toolbox  along.  A  method  of  ruvigation  that  is 
the  complement  of  flying  is  “grabbing"  the  world.  Crabbing 
the  world  allows  the  user  to  attach  the  modeling  space  to  the 
cursor  and  then  drag  and  route  iL  Grabbing  can  be  used  to 
bring  a  feature  of  the  world  to  the  user  rather  than  forcing  the 
user  to  walk  or  fly  to  the  feature. 

Models  often  require  manipulation  at  vastly 
different  scales.  To  facilitate  this  type  of  work,  the  user  can 
be  scaled  using  s  pnsccss  cslled  growing  and  shrinking.  This 
scaling  does  not  affect  the  model:  it  changes  the  user's 
relative  siu  with  respect  to  the  model.  The  user  could  shrink 
down  to  bird  size  in  order  u>  add  eyelashes  to  a  model  of  an 
elephant  and  then  grow  to  the  size  of  a  house  to  alter  the 
same  model's  legs.  Since  the  user  can  become  disoriented  by 
all  of  these  methods  of  movement,  there  is  a  eommand  that 
immediately  returns  the  user  to  the  initial  viewpoint  in  the 
middle  of  the  modeling  space. 

The  user  receives  continuous  feedback  in  a  variety 
of  ways.  The  HMD  system  provides  all  visual  input  to  the 
user,  so  the  display  must  be  ujxiaied  between  IS  and  30  times 
per  second.  Even  during  file  loading  end  other  slow 
operations,  ihe  screen  is  updated  end  the  head  is  tracked. 
Rubber  bending  is  implemented  in  many  situations:  when 
defining  •  new  triangle,  sealing  or  moving  an  object,  and 
extruding.  Predictive  highlighting  shows  the  user  what 
would  be  selected  if  a  mouse  button  were  pressed.  This 
highlighting  is  used  in  the  toolbox,  and  even  more 
imponantly.  when  marking  vertices.  Whenever  the  cursor  is 
near  a  m^el.  the  nearest  vertex  is  highlighted,  giving  the 
user  an  indication  of  which  vertex  would  be  operated  on 
before  actaaDy  auempeing  the  iteration. 


Figure  1:  The  toolbox  as  saw  by 

The  toolbox  initially  appears  suspended  in  space 
the  user's  a  more 

convenient  location.  The  toolbox  remains  aoechad  to  the 
user  M  he  or  she  moves  vound  the  modeling  space,  or  can 
be  disconnected  end  left  m^hcrc  above  the  magic  carpet. 

The  toolbox  cells  containing  3D  icons. 

Each  icon  represents  either  ■  a  command,  or  s  toggle.  * 

Many  these  icons  can  optionally  appear  in  pulldown 
at  the  lop  of  the  toolbox  order  reduce  chiucr. 

perform  when  they  arc 

the  cursor.  change  the  current  mode  of  operation  as 

the  shape  the  cursor.  For  instance,  when  the 
user  reaches  inui  the  toolbox  and  selects  Ihe  flying  tool,  the 
cursor  takes  the  form  of  an  airplane.  Selecting  a  command 

performs  a  single  task  without  changing  the  current  mode  of  Figure  2:  A  triangle  being  added  to  a  model.  Demonstrates 

rubber  banding  and  snapping  to  nearby  vertices. 


3.2  Tools  and  Commands 

Ahhough  many  tools  are  available  in  the  3dm 
toolbox,  it  is  more  useful  to  understand  the  general  classes  of 
tools  supplied  to  the  user  than  to  «iumeraic  all  of  the  spocific 
tools.  Most  of  these  tools  were  chosen  because  of  (heir 
proven  utility  in  pre-existing  modelers. 

3.2.1  Surface  Creation 

Surface  creation  is  the  central  purpose  of  most  3D 
modeling,  so  3dm  provides  more  ihu  one  method  for 
creating  surfaces.  A  triangle  creation  tool  exists  for 
generating  both  single  triangles  and  triangle  strips.  The 
comers  of  these  triangles  are  specified  by  pointing  and 
clicking  the  mouse,  so  the  triangles  me  created  in  their 
desired  locations  rather  than  appearing  in  a  "building*  area 
and  then  being  moved  into  the  model  space.  Pre-existing 
vertices  may  be  used  during  triangle  creation  to  allow 
triangles  to  share  comers  or  entire  edges,  making  seamless 
connections  easy. 

The  extrusion  tool  supplies  a  more  powerful  and 
more  specialixed  method  of  triangle  creation.  This  tool 
allows  the  user  o  either  draw  a  poly-line  or  select  one  from 
edges  already  in  the  model  and  stret^  it  out  into  m  extruded 
surface.  The  extrusion  is  performed  by  dragging  the  leading 
edge  of  the  stvface  with  the  mouse.  Becanse  the  mouse  cat  be 
twisted  and  translated  arbitrarily  during  the  extrusion,  it 
becomes  easy  to  create  complex  surfaces  with  this  tool.  In 
additioru  the  leading  edge  of  this  new  surfaec  can  be  scaled 
and  then  extruded  again  as  many  times  u  necessary.  This 
form  of  extrusion  can  rapidly  create  such  objeca  as  walls, 
legs,  tree  trunks,  and  leaves. 

The  lut  surftMte  creation  tools  ficilitaia  creation  of 
standard  surface  shapes.  Currently  box.  sphere,  and  cylinder 
looU  exist.  They  csch  allow  the  user  u>  inicractivcly  stretch 
out  sn  arbitrarily  proportioned  wireframe  repreaenuiion  of  a 
standard  shape.  Whm  the  wireframe  representation  has  the 
desired  proporuons,  it  is  turned  into  a  triangulated  surface. 

3.2.2  Editing 

Since  surfaces  are  rarely  in  exactly  the  desired  shape 
upon  creation,  it  is  important  thm  surface  editing  be  m  easy 
operation.  The  most  commonly  used  editing  tool  is  the 
mark/movc  tool  This  tool  provides  a  method  of  grasping 
and  moving  arbitrary  portions  of  the  model  Not  only  can 
cntiia  objects  be  grafairnd  and  moved  with  the  mouse,  but 
selected  groups  of  vertices  can  be  moved  in  order  to  distort 
pan  of  an  object  Scaling  can  also  be  performed  on  either 
entire  objects  or  groups  of  verticea.  During  both  movement 
and  scaling,  the  user  secs  the  model  changing  in  real  time. 
Thia  interaction  decreases  the  number  of  edits  needed  to  make 
a  desired  change.  The  marking  aspects  of  this  tool  ae  used  to 
mvk  arbitrvy  portions  of  the  model  for  operations  erith 
ether  tools. 

Familiv  editing  operstions  from  drawing  programs 
tre  ■  group  of  3dm  commands  that  facilitsie  rapid 
cxpehmenial  changes.  An  vbiusry  number  of  triangles  or 
entire  objects  can  be  cut  copied,  pasted  and  deleted.  These 
commands  provide  easy  reuse  of  existing  objects. 

An  undo/redo  stack  is  provided  for  reversing  any 
number  of  operations  from  any  tool  or  command.  As 
operations  are  performed,  the  changes  they  cause  to  the 


model  arc  stored  in  the  undo/rcOo  stack.  The  undo  command 
can  then  be  used  to  pop  changes  off  of  this  stack  to  undo  as 
many  operations  as  necessary.  These  undo  operations  can 
themselves  be  undone  with  the  redo  command.  The  undo/reJo 
commands  encourage  experimental  changes  to  the  model 
because  no  operation  can  cause  permanent  damage. 

3.2.3  Hierarchy 

The  hierarchical  features  of  3dm  provide  methods 
for  organizing  complex  models.  “Grouping"  can  be  used  to 
associste  triangles  and  possibly  other  groups  to  more  easily 
manipulate  them  as  a  whole.  These  groups  can  be  itiaanced. 
An  instance  ia  similar  to  a  copy  of  a  group  that  can  be 
arbitrarily  translated,  routed,  and  scaled.  However,  the 
difference  between  an  instance  and  a  copy  is  that  the 
insunoes  of  a  group  art  all  linked  us  the  same  basic  shape.  If 
this  shape  is  changed,  then  the  change  is  raflecud  in  all 
instances  ai  once.  An  example  where  instancing  would  be 
uaeful  is  ia  e  model  of  a  large  building.  Suppose  that 
hundreds  of  chairs  were  in  this  building.  If  one  model  of  s 
chair  were  instanced  many  times  to  make  these  chain,  then  a 
change  to  a  single  chair  would  be  reflecud  in  hundreds  of 
places  throughout  the  building. 

Croups  can  be  organized  into  s  hierarchy 
represented  by  s  directed  acyclic  graph.  This  type  of 
himarchy  is  particularly  well-suited  to  modeling  articulsied 
Tigures.  The  ability  to  instance  groups  and  impose  a 
himarchy  on  them  helpa  to  organize  models. 

4  Results 

Actual  modeling  sessions  have  shown  that  3dm  is 
cfTicieni  for  rapidly  prototyping  models.  Organic  shapes, 
like  racks  and  trees,  have  proven  to  be  particularly  good 
subjecu  for  3dm.  These  shapes  are  easily  aeaud  in  3dm 
because  it  provides  a  good  sense  for  spatial  relationships. 
Users  of  3<lra  have  commented  that  tluy  feel  a  sense  of 
control  because  they  can  reach  out  and  grab  any  part  of  the 
model  with  ease.  The  ability  to  make  these  quick 
modificaiions  encourages  the  user  to  experiment  with  shapes 
until  they  arc  satisfactory.  However,  3dm  has  shown 
weakness  in  the  area  of  constraints  and  models  that 
traditional  CAD  and  drawing  programs  creau  well.  For 
instance,  3dm  has  no  way  of  keeping  two  polygons  parallel, 
causing  some  models  to  appear  irr^ular. 

The  extrusion  tool  ia  an  example  of  a  traditional 
modeling  tool  that  has  become  even  more  powerful  because 
of  iu  use  in  a  HMD  framework.  In  most  modeling  systems, 
extrusion  is  performed  by  moving  one  or  two  spatial 
parameters  at  a  tune.  3dm  uaers  often  alter  many  parameters 
at  once  ihving  an  extrusion  by  twisting  end  irandsting  the 
new  surface.  Extrusion  in  3dm  often  consists  of  msny  short 
extrusions.  In  between  these  short  operstions  the  leading 
edge  of  the  extruded  surface  is  often  scaled  and  twisted.  The 
result  is  that  complex  surfaces  can  be  rapidly  created  with  an 
aaay  to  use  tool. 

Some  initial  solutions  to  3dm 's  lack  of  consuainis 
have  baen  to  add  urggles  in  the  toolbox  for  a  snap-to  grid  and 
a  snap-io  plane.  The  tnap-io  grid  constrains  the  position  of 
the  cursor  to  the  nodes  of  s  regular  3D  grid.  The  resolution  of 
the  snap-io  grid  is  dynsmically  modined  to  be  appropriate  to 
the  user's  current  “grown"  or  “shrunk"  sire.  The  snsp-to 
pisne  gives  the  ability  to  constrain  cursor  movement  to  2 


diinensiofu.  The  map-io  coiuireinu  help  in  making  regular 
ohjecu.  luch  as  mechanical  parts 

5  Conclusion 

3<im  draws  techniques  of  model  manipulation  from 
both  CAD  and  drawing  programs  and  applies  them  lo 
modeling  in  an  inuiiive  way.  A  ilMD  modeling  system  uses 
these  tools  to  simplify  the  problem  of  30  model 
manipulation  and  undmtsnding. 

3dm  is  a  step  toward  making  30  modeling 
accessiUe  to  tmaophisticated  users.  It  supporu  users*  natural 
forms  of  interaction  with  objccu  to  give  them  beuer 
understaading  of  the  abapes  of  iheiT  models.  Even  a  novice 
user  can  understand  how  to  manipulau  a  model  by  reaching 
out  and  grasping  it.  Users  an  encouraged  to  enperimam  with 
nwdal  shape  bseausa  3dm  facilitates  making  rapid  changes. 
Hie  effects  of  a  change  to  a  model  can  ba  ckarty  understood 
because  the  user  can  explore  the  model  using  a  variety  of 
intuitive  aavigatkm  techniques. 

Advanced  users  arc  also  empowered  by  3dm.  Many 
of  the  tools  borrowed  from  existing  modeling  systems 
become  more  powerful  when  used  with  a  HMD.  One  aotnee  of 
increued  utility  is  the  fact  that  complex  operations  can 
involve  simultaneous  raodification  of  many  spatial 
parameters.  Examples  of  tools  that  take  advantage  of  this  are 
object  placement  and  extrusion,  which  both  allow 
combinations  of  rotation  and  translation  in  a  single  step. 
By  concentrating  more  fimcdonality  into  each  operation. 
few«  operations  ara  needed  to  perform  a  task  and  models  can 
be  creat^  faster. 
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ABSTRACT 

A  oonoolkd  experimeni  wu  conducted  to  compac  bead-txacked 
■id  non-head-mcked  stocrint  modes  in  the  p^onnaaoe  ct  an 
abstract  beam-targedni  task.  CoUecteddauievealedaaddevsiety 
of  mode  preferences  among  the  subjects.  Subjeoperfoiiaaioe,  as 
measured  by  final  scoie.  task  completion  time  end  subject  confi¬ 
dence,  differed  very  little  between  the  head-tracked  steering  modes 
taken  as  a  group  and  the  collective  non-head-iraeked  modes.  Some 
significant  differences  etoe  observed  between  individual  steering 
modes,  both  arithin  «id  betamen  the  head-tracked  and  non-bead- 
tiacked  groups. 

MTROOUenON 

Current  lesearch  at  the  Univenity  of  North  Carolina  «  Qiqtel  Hill 
is  investigsting  the  possible  benefits  to  be  gained  by  epplyk^  bead- 
mounted  display  (HMD)  technology  to  radiotherapy  treaanem  plan- 
ningCRTP}.  Use^ahead-moumed display  for  targ^g of  neatment 
beams  suggesu  several  possible  steering  modes  for  exploring  the 
virtual  world  of  the  paiietu's  aruuomy.  Todetennine  wh^  steering 
mode  is  best  suited  to  our  application,  a  user  study  was  conducted  to 
investigate  the  relative  merits  of  the  different  steering  modes. 

Seven  steering  modes  were  used  in  an  absoaa  beam-targeting  task. 
Four  modes  vued  head-tracking  mformaiion,  while  the  other  three 
modes  did  not.  It  wu  anticipated  that  head-tracking  would  provide 
an  advantage  in  beam  targeting  through  more  natural  steering  arul 
navigation  that  makes  use  of  proprioceptive  and  vesu'bulv  ntfatma- 
tion.  which  are  absent  in  nen-head-tneked  methods. 

Related  work  in  movement  through  a  vntual  world  is  described  in  [  1 , 
2. 3],  but  these  studies  do  not  deal  with  HMD's  and  head-tracking. 

BEAM  TARGETING 

The  key  to  successful  beam  targeting  in  radiation  therapy  treatment 
planning  is  to  orient  and  shape  the  beams  so  that  the  entire  tumor  is 
covered  by  each  beam  while  as  little  of  the  healthy  surrounding  tissue 
as  pouible  is  hit  by  the  beams.  Given  the  complex  spatial  arrange¬ 
ment  of  a  patient's  anatomy  (tumors  may  be  draped  around  healthy 
orgaiu  or  have  tendrils  snaking  out  into  the  healthy  tissueX  this  is 
usually  not  an  easy  task. 

To  evaluate  the  difTerem  Meeting  modes,  subjects  were  presented 
with  an  abstract  anatomy  model,  coruisting  of  a  multi-oolorod 
sphericai  tar  get  ( tumor  analog)  em  bedded  in  a  coOection  of  uniqudy 


eolcrod  munfaJiromatic  spherical  dodges  (organ  analogs).  (See 
ColorPIatol.)  The  subjsca  used  eadi^  the  seven  steering  modes 
to  manipolaie  tbe  direedan  in  which  a  conical  virtual  beam  passed 
through  tbemodeL  Ihe  beam  was  defined  such  that  its  source  (cone 
vertex)  wm  alwiys  a  fixed  distance  from  the  urget,  its  central  ray 
(cane  axis)passedihrought  the  center  of  tbe  target,  andits  divergence 
was  just  large  oiough  to  encompass  the  target  The  subject  was 
to  find  tbe  beam  directian  that  afforded  the  smallest 
votume  of  intenectaan  between  the  ooaiical  beam  ind  the  dodges,  a 
task  analogous  to  a  ndiothcr^riat  trying  to  avoid  radiosensitive 
organs  arith  a  Bcaonent  beam. 

STEERMGMOOES 

‘Ibe  term  “saaering  mode**  refers  to  the  method  used  to  change  one  ‘  t 

ynritirw  Qf  ocicntatian  in  the  virtual  world.  This  is  dininguished 
from  navigndon,  which  refen  to  undentmding  one's  current  posi¬ 
tion  and  crientaiicn  relative  to  other  objects  in  the  vinual  world. 

HnndTfnctod 

These  modes  are  linked  to  movement  of  the  subject 's  head  and  enable 
the  subject  tomake  use  of  vestibular  (inner  ear  balance)  snd  proprio¬ 
ceptive  (muscles,  undoiu,  joous)  senses  for  navigation. 

HMtoroundOUX).  In  Walkaoond  mode,  the  subject  physically 
aralks  about  in  Ihe  vinual  world  oontaming  the  target/dodges  model. 
Ibe  diroction  of  ihe  beam  it  defined  by  tbe  vector  from  the  subject 's 
eyes  to  the  center  of  the  target  To  better  examine  the  model  and 

target  the  beam  from  above  and  below,  tbe  subject  it  given  the  ability 

to  vertically  translate  the  model  using  ■  6-D  mouse.  No  other 
msnipulatian  of  the  model  is  possible. 

Wttarourd/noMhnCWKR).  ThisisthesameatWalkBoundmode. 
eriik  the  exception  that  the  subject  is  able  to  also  route  the  model 
aboot  any  axis  in  3-space  ihroogh  its  center  by  grabbing  with  the  6- 
0  mouse.  (See  6-D  Mouse  tecdon  below.) 

Ortital(ORB).  In  Orbital  mode  the  subject  is  constrained  to  alwiys 
be  lod^g  at  the  center  of  the  model  from  the  beam  source.  Beam 
doection  eoincidea  with  gaze  direction.  Unlike  the  Walkaround 
modea.  Orbital  mode  uses  only  head  orienution  and  ignores  head 
poaitioo.  As  the  subject’s  bead  turns,  the  model  is  observed  to 
tmtslate  about  the  subject’s  bead  at  a  constant  distance.  (Hence  the 
nme  OibiiaL)  Because  the  model  undergoes  no  rotation,  it  can  be 
viewed  from  any  direction  with  a  turn  of  the  subject's  head. 

tnirmthn  (UtiM).  In  Immenion  mode  the  subject  views  the  model 
looking  outward  from  the  center  of  the  target  Like  Orbital  mode, 
Immenion  mode  makes  use  of  head  orienution  only  and  ignores 
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head  poaition.  When  the  mbjeo'a  head  nnia.  the  aibjea'i  view 
fweepa  acrou  portions  of  the  model  from  ha  fiaed,  oeanl  veniage 
poiiu.  The  beam  direction  is  defined  by  the  subject's  gaze  direction, 
and  the  task  of  finding  the  best  beam  oriemaiian  becomes  one  of 
'  looking  for  the  portion  of  the  model  wiiblhe  biggest  opening.  Since 
the  beam  passes  completely  through  the  model,  the  subjeo  is  given 
the  ability  to  reverse  his  geze  directian  by  holding  down  a  butiOT  on 
the  6-0  mouse,  and  can  thereby  examine  the  complete  prospective 
beam  path  through  themodcL 

NeivHMel>Tracfcdd 

Although  these  modes  do  not  make  use  oTbead-traclonginfannaiian. 
the  subjectt  still  viewed  the  model  through  the  HMD  so  that  image 
rpiality  was  equalized  over  the  seven  modes.  These  three  modes  all 
place  the  subjeo’s  eye  at  the  beam  soistoe,  looking  in  the  directian  of 
the  beam  trrward  the  target,  and  support  exploration  of  prospective 
beam  orieruatiatu  by  rotating  the  model  in  three-space. 

InJoystickmodethemodcIisiaiaiedwiihavelocity- 
conirol  joystick.  In  addition  to  the  left-right/rarward-backward 
movemeru  of  the  joystick,  the  cap  of  the  joystick  lunu  clockwise  and 
cowuerclockwise  to  provide  an  three  degrees  of  rotadonal  fteedom. 

Spae»ball(SPC).  InihismodeihemadelisroiaiadarithaSpaoeball*. 
an  isometric,  force-setuitive  device  that  provides  six  d^rees  of 
tnmslstionalarKl  rotational  frrndnriL  This  trrode, however,  uses  only 
the  three  rotational  degrees  of  fieedom  as  a  velodiy  control  for 
rotation  of  the  model  in  three-space. 

$-0  Moub0  (SDM).  In  6-D  Mouse  mode  the  orieruation  of  the  tttodel 
is  controlled  with  a  custom-built,  six  degree-of-Creedom  mouse 
(oeckCT  sensor  embedded  in  a  pool  ball  with  two  buttons).  When 
either  mouse  button  is  held  do^  the  rotational  componertt  of  the 
ntouae  movement  is  directly  linked  to  model  rotation,  and  the  subject 
tees  the  model  rotate  in  the  tame  manner  as  his  hand. 

BEAM'S-EYEVIEW 

An  important  feature  of  a  steering  mode  that  may  affea  a  subjea't 
performance  it  whether  or  not  it  provides  a  "beam's-eye  view." 
Beam 's-«ye  view  is  the  view  teen  by  an  eye  coincident  with  the  beam 
vertex  and  whose  gaze  vector  coincides  with  the  beam’s  cerural  axis. 
With  a  beam's-eye  view  it  is  very  easy  to  determine  which  dodges 
intersect  the  beam,  for  since  the  bem  is  defined  to  divoge  just 
enough  to  exactly  enclose  the  target,  the  silhouenes  of  those  dodges 
will  overlap  with  the  silhouette  of  the  target  In  those  modes  that  do 
not  provide  a  beam's-eye  view,  it  is  more  difficult  for  the  subjea  to 
judge  which  dodges  are  hit  by  the  beam. 

Walkaround  and  Walkaround/Rotaie  modes  donor  provide  beam's- 
eye  views,  because  the  subject’s  head  eannoi  be  physicaily  oon- 
so’ained  to  align  with  the  beam  source.  Immersion  m^  also  does 
not  provide  s  beam's-eye  view,  since  the  subject's  eyepoim  is  con¬ 
strained  to  suy  at  the  target's  center.  The  ocher  four  modes  do 
provide  beam's-eye  views. 

EXPERIMENTAL  METHOD 

The  experiment  was  a  one-factor  within-subject  investigation,  with 
steering  mode  u  the  independent  variable.  Dependent  vatiaUes 
measured  were  final  score  (volume  of  inieisection  between  beam  and 
dodges),  task  completion  time,  confidence  in  the  final  beam  configu- 
raiion,  and  rank  orderinp  of  the  seven  modes  by  ease  of  use  and  by 
preference. 

Fourteen  subjects  were  recruited  from  graduate  snidenB  and  staff 
members  of  the  Deparrmenu  of  Computer  Scienee,  Radiation  On- 

'Spaceball'™  is  s  registered  trademark  of  Spatial  Systems,  Inc. 
'3Spacc^  is  a  registered  trademark  of  Polhemus  Navigation  Sci- 
eitces. 

•EyePhone''*  is  a  registered  trademark  of  VPL  Research.  Inc. 


eology.  and  Radiology  at  UNC.  Each  subject  underwent  7  sessioiu. 
each  of  which  used  a  different  steering  mode.  The  order  of  the 
steering  modes  used  by  each  subject  was  varied  according  to  a  7x7 
laiin  square.  Each  session  consisted  of  3  practice  trials  followed  by 
3  lest  trials.  Each  trial  used  a  unique  target/dodge  model. 

In  each  trial  the  subject  explored  prospect!  ve  beam  orientaiioiis  tin  ti  1 
the  best  oik  was  found,  at  which  time  the  subject  stopped  the  trial. 
There  was  no  liine  limit,  nor  any  emphasis  on  task  completion  time — 
Ihesubject  was  instrucied  to  take  as  long  as  necessary  to  find  the  best 
beam  path.  A  vimial  marker  (arrow  pointing  through  the  model)  was 
pnviM  to  the  subject  muse  as  a  reference.  At  any  time  the  subject 
could  issue  a  "mark"  command,  which  aligned  the  marker  with  the 
euncat  beam  directico,  and  the  marker  would  rentain  fixed  in  the 
model  until  a  subsequent  command  was  issued.  The  score  and  task 
conqiletion  dme  for  the  Dial  were  recorded,  as  well  as  the  subject's 
ruing  on  a  scale  of  1  (no  confidence)- 10  (total  confidence)  of  how 
confident  he  or  she  was  that  the  bat  beam  orientation  had  been 
fbimdL 


After  all  seven  sessions  were  completed,  the  subject  rank  ed  the  seven 
steering  modes  according  u>  two  criteria,  ease -of-use  of  the  steering 
mode  and  preference  for  performing  die  beam  targeting  task. 

Equiproem  used  included  a  Polhemus  3Space'  tracker  on  in 
Eycnione*  Model  2  head-mounted  display,  displaying  images  gen- 
erued  by  UNC's  Pixel-Plines  4  grsphics  piocessor. 

RESULTS 

Rgure  1  presents  histograms  showing  for  each  steering  mode,  the 
number  of  timea  it  was  ranked  1st,  2nd.  —  7th  by  ease -of-use  and  by 
prefcrenix.  Theplot  for  Walkaround  Mode  shows  that  most  subjects 
found  it  to  be  one  of  the  more  difficult  steering  modes  to  use.  The 
other  three  head-tracking  modes  have  somewhat  flu  histograms, 
suggesting  no  general  consensus  on  how  euy  they  were  to  use.  Of 
the  non-head-cracking  modes,  the  Joystick  mode  is  widely  consid¬ 
ered  an  easy-to-use  steering  mode.  Spaceball  mode  and  6-D  Mouse 
mode  both  tended  to  be  on  the  difficult  side. 


The  preference  rankings  show  ihu  Joystick  mode  wu  widely  pre- 
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ferred  by  the  wbjects,  whcnai  Wilkvound  Mode  vm  widely 
disliked.  For  Spaoebell  and  6-D  Mouse  inodes  subjects'  opinions 
were  on  (he  less>prefened  sides.  Ontheotherhand,Wilk/Rotateand 
Orbital  modes  were  on  the  more-preferred  side  of  the  scale.  Immer¬ 
sion  mode  shows  an  interestini  bimodal  distnbuiioa,sugxesting  that 
subjects  either  loved  it  or  hated  iL 

In  order  to  factor  out  mter -model  variability,  each  trial's  score  was 
normalized  by  the  median  score  across  subjecu  for  the  particular 
model  used  in  that  trial  Figure  2  shows  the  distribution  of  the 
logarithms  of  the  normalized  scares  grouped  by  steering  mode.  As 
the  beat  possible  score  is  0  (no  intersectian  between  beam  and 
dodges),  the  more  negative  values  repiesem  better  performance. 
Student's  t-test  reveals  significant  differences  for  the  following 
inter-mode  comparisons:  IMM-ORfi  (oasO.CVjdT),  IMM-SPC 
(a*0.0069).SDM-ORB(owO.Ol26).WK*'  T  J(a»Of)irr).IMM- 
JOY(a«0.0197).  Head-trackedmodesd*  M.ORB.WKR,  WLK) 
taken  as  a  group  do  not  differ  sign'  .c  ’  / horn  nan-head-tracked 
modaUOY.SDM.SPC). 

Figure  3  shows  the  distributiort  of  the  logarithms  of  the  task  cookie- 
lion  limes  grouped  by  steering  mode.  No  stgnincam  diffaences  are 
found  in  this  data,  neither  between  individual  steering  modes  nor 
between  head-irackeu  and  non-head-tracked  modes. 

Figure  4  piesents  the  distribution  of  subject's  conTidence  rating 
grouped  by  steering  mode.  The  only  significant  effect  found  in  this 
dau  is  the  JCY-IMM  comparison  (ocs0.037S}. 

Table  1  describes  correlations  between  the  dependent  variables.  Not 
surprisingly,  ease-of-use  and  preference  rank  are  highly  conelated. 
Significant  correlatioru  are  alM  found  between  subject  confidence 
and  ease-of-use,  preference,  and  elapsed  time.  AH  three conrelations 
are  negative,  indicating  that  a  subjects’  confidence  decreased  when 
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using  diHicult  steering  modes  or  modes  they  did  not  like,  or  when 
trials  took  a  kmg  time.  Interestingly,  score  is  not  significantly 
conelated  with  any  of  the  other  verifies. 

DISCUSSION 
Trial  Roplay 

In  addition  to  the  statistical  summaries  presented  above,  a  subject!  ve 
review  of  each  trial  was  conducted  by  playing  back  log  files  in  which 
were  recorded  status  infannatian  for  the  subject's  head,  the  model 
and  the  beam  at  half-teoond  intervals.  By  observing  the  trial  replay 
«dth  the  HMD,  it  was  possible  to  study  how  the  subject  moved  snd 
how  the  model  wu  manipulated.  T>k  trial  r^lay  also  triced  the 
location  of  thebeam  source  through  the  trial  quickly  revealing  which 
beam  disectioni  were  considered,  and  peihaps  more  important, 
which  directions  were  not  considered. 

In  spite  of  being  instructed  to  find  the  best  possible  beam  direction, 
subjects usuaDy  teiminaied  the  trial  before  considering  all  possibili¬ 
ties.  Presumably  they  were  able  to  attain  a  good  enough  spatial 
understanding  of  the  model  edthoui  hiving  to  inspect  it  from  all 
angles.  Trials  in  which  the  model  had  been  completely  covered 
usuaOy  srere  usually  of  extremely  long  duraDon,  with  subject  move¬ 
ment  suggesting  confusion  and  disorientation.  In  only  a  few  cases 
did  subjects  follow  a  systematic  search  strategy,  and  these  systematic 
searches  would  usu^y  be  abandoned  after  one  candidate  beam 
direction  had  been  found.  For  the  most  part,  subjects  followed  what 
might  be  called  a'*greedy’' steering  strategy ,  moving  about  the  model 
in  amanner  based  upon  their  current  view  of  the  model,  and  not  upon 
some  predefined  plan.  As  s  result,  in  most  trials  the  traces  of  the  beam 
source  showed  large  "holes"  that  were  never  considered.  From  just 
watching  the  trial  replay  it  is  difficult  to  determine  whether  such 
holes  were  areas  that  were  deliberately  skipped  or  accidentally 

missed.  Some  of  these  areas  corresponded  to  beam  directions  that 

were  obviously  bad,  which  might  imply  that  those  ponibilities  were 
deliberately  slopped.  Otherholes  contained  prospective  beam  direc- 
dora  that  were  good  aiough  to  deserve  consideration,  implying  that 
these  areas  were  accidentally  missed  by  the  subject.  In  most  cases  the 
beam  directions  that  required  the  siibject  to  look  straight  up  or 
straight  down  were  not  covered,  as  the  HMD  would  exert  very  large 
uaques  on  the  subject's  neck  in  these  positions. 

Moat  subjects  relied  very  heavily  on  the  marker  to  provide  a  refer¬ 
ence  point  in  the  model.  The  marker  served  as  a  landmark  that 
facilitated  quick  movement  between  two  diametrically  opposed 


beam  directioiis,  and  was  also  typically  used  as  a  **besi-beam- 
diieciion-io-far'*  marker  to  which  the  subject  would  reaim  for  the 
final  solution  after  furtherexplontian elsewhere.  Many  subjecu ex¬ 
pressed  a  desire  to  have  more  than  one  marker.  Most  subjects  did  not 
make  useof  the  context  provided  by  dodges  laiiquely  colored  inHLS 
space  for  reference.  Only  one  subject,  whose  own  research  is  con¬ 
cerned  with  the  use  of  color,  fotmd  the  colon  useful-^  useful  in 
fact,  that  the  marken  were  never  used. 

StMrIng  Mod*  Summarto* 

Imawnion  (IMM).  Immersion  mode  prediKed  signiTicantly  worse 
scores  than  Orbital.  Spaceball.  and  Jcqrstick  modes,  aad  it  appeared 
to  have  instiUed  less  confidence  in  the  subjects  than  the  otherraodes. 
This  may  be  a  result  of  the  subjects'  being  able  to  see  only  a  «m«ti 
portion  of  the  model  at  any  time,  which,  oombined  with  the  lack  of 
any  head-motion  parallax,  eoold  have  hindend  tte  subject's  devel¬ 
opment  of  a  complete  mental  picture  of  the  "vvtgt  In  addttion. 
subj^  were  required  to  evahiaie  prospective  beam  orientataons  by 
looking  in  one  direction  md  then  in  the  other  ditection.  with  no  clear 
indication  of  where  the  boimdaty  of  the  beam  was.  Ifflinerskin  mode 
did.  however,  have  the  advantage  of  providing  the  ability  far  the 
subject  to  use  muscle  memory  in  navigation.  Even  erithout  a 
oompiete  global  understanding  of  the  model,  subjects  Imew  how  they 
had  to  orient  their  heads  to  get  back  to  a  partieulv  beam  dnection. 

Orbital  (ORB).  Despite  the  fact  that  there  is  no  real-world  metaphor 
for  this  steering  mode.  Orbital  mode  produced  significantly  better 
scores  than  Immersion,  6-D  Mouse,  and  Walh/Rotate  modes.  This 
may  have  been  due  u>  the  unique  combination  of  several  factors. 
Orbital  mode  provides  s  beam's-eye  view  of  the  model,  which  at 
once  gives  the  subject  n  external  global  view  of  the  mfMkI  and 
allows  the  subject  to  easily  determine  which  dodga  intersected  the 
beam.  Another  contributing  factor  is  the  aid  to  navigatian  through 
muscle  memory  provided  by  Orbital  mode. 

Walharounb  fWLX)-  Wtlkaround  mode  produced  the  longest  mean 
task  completion  time,  but  was  undistinguished  in  score  and  subject 
conTidence.  The  long  task  completion  time  is  not  surprising,  given 
the  difficulty  of  waltog  about  in  the  virtual  world  in  a  Hl^  that 
seals  off  any  view  of  the  real  world.  Most  subjects  found  this  »««<*• 
very  swkwird  and  time-consuming,  and  ranked  Walkaoimd  low  in 
ease  of  use  and  preference.  Interestingly,  this  mode  more  then  any 
other  wu  used  for  systematic  searches.  One  subject  repeatedly 
circled  around  the  model  inspecting  the  model  at  difTerent  heights 
with  each  loop.  Another  subject  opted  to  walk  less  and  inspect  the 
model  verbcsily  at  regular  intervals  vottnd  the  model  Perhaps  the 
awkwardness  of  the  mode  msiilled  in  these  subjects  a  need  far  a 
disciplined,  efTicieni  approach. 

Walkanuml/Rotation  fWKHj.  Walk/Rotate  mode  did  not  perfonn 
any  beuer  than  Walkaround  mode,  but  fared  better  in  ease  end 
preference  rankings.  The  model  rotation  capability  was  used  to 
different  degrees  by  the  different  subjects.  Most  subjects  walked 
very  little  nd  spent  most  of  their  time  standing  s&Q  and  rotating  the 
model  as  in  6-D  Mouse  mode.  Some  triab  showed  no  rotation  at  all, 
perhaps  indicating  a  relucunce  in  the  subjea  lo  lose  the  navigational 
advantage  provided  by  a  fixed  model  reference  frame. 

JoyateM  (JOY).  Joystick  mode  ranked  very  high  in  caK-of-tise  ani 
preference,  probably  because  most  of  the  subjects  worked  with 
computers  and  were  somewhat  famOiar  with  video  games.  Even  so, 
perfomience  with  Joystick  mode  wes  not  notable.  Trial  replay 
revealed  that  most  subjects  used  only  principal  axis  rotations,  i.e. 
they  roiaiedmodels  mmtly  venicaUy  and  horizontally  ind  very  little 
disgonally.  This  was  protably  due  to  the  mechanical  action  of  the 
joystick,  which  requii^  slightly  more  cfTort  to  move  diagonally. 
The  efTeci  of  this  restriction  is  unclear,  for  while  it  forced  subjects  to 
decompose  their  movemenu  into  a  series  of  principal  axes  rotations. 
It  provided  a  precision  of  movement  not  avaO  tble  with  the  othemon- 


head-iracked  modes. 

6-0  Mouaa  (SDM).  Compared  to  subjects'  preference  for  Joystick 
mode  and  dislike  for  Walkaround  mode,  response  to  6-D  Mouse 
mode  was  relatively  flat.  lu  performance  was  undistinguished  from 
the  other  modes.  Trialreplsysshowedthatthismodesufferedgreaily 
fiom  cracker  latency,  which  greatly  hindered  both  i^ecise  alignment 
and  movcnMxUB  large  enou^  to  require  more  than  one  gnb-release 
cycle.  Consequently,  beam  source  traces  for  6-D  Mouse  mode  were 
chmeterized  by  a  very  jagged  appearance  with  large  direction 
changes  trpgiang  relatively  small  rotations. 

Sfiaeabat(SPC).  The  perfannanoe  of  SpacebsU  mode  is  relatively 
undistinguished,  but  ia  prefemce  rankings  are  weighted  toward  the 
low  csd.  Many  subjeeu  found  the  SpacebaU  fatiguing  and  difficult 
10  oae  for  precise  inovtmesits. 

Ocnaril  Cemmants 

Pethapemoct  compelling  is  the  large  imer-subjea  ventnee  seen  this 
experimou,  which  mey  have  muked  significant  differences  be¬ 
tween  aeering  inodes  nd  between  the  collective  head-tracked 
nodea  and  the  non-head-oicked  modes.  Standardized  tests  of 
^atial  crioitatian  and  spatial  vitualizadan  [4]  may  provide  a  nor¬ 
malizing  factor  to  reduce  this  variance. 

Another  interesting  observation  is  the  large  variation  seen  in  the  pref- 
crencemdease-of-usehistogramsofthehead-nckedmodes.  There 
waa  no  general  consensus  about  which  of  the  four  modes  was  the 
best,althDughWalkaroundwBt  generally  considered  the  worst.  This 
soggesu  that  to  be  widely  accepted,  an  HMD-based  targeting  tool 
should  have  an  ad^xable  user  inierface  that  lets  users  choose  the 
steering  mode  they  want  to  use.  One  must  also  consider,  however, 
that  a  task  so  critical  as  targeting  of  treatment  beams  demands 
opdmal  performance.  Orbital  mode  provides  bener  performance 
than  the  other  three,  and  will  be  carried  ova  in  to  the  ne  X  t  experimen  t . 
involving  true,  anatomical  beam-targeting.  Since  score  is  not  corre¬ 
lated  with  mode  preference,  it  is  expected  that  performance  of  users 
who  do  not  like  Orbital  mode  w'lll  not  suffa  from  having  to  use  it. 

CONCLUSIONS 

Collected  dau  show  no  significant  difference  between  head-tracked 
steering  modes  and  non-head-  tracked  steering  modes  in  the  perform  - 
ance  of  an  abstraa  beam-tageting  task.  Orbital  mode  provided  the 
best  overall  performance.  Immersion  mode  the  worsL  The  three  non¬ 
head-tracked  modes  were  not  distinguished  by  perfoimance. 
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ABSTRACT 

In  a  vinual  world  viewed  with  a  haad-noontad  display,  tfaa  user 
nuy  wish  to  perform  certain  actions  mder  the  control  of  a 
manual  inpat  device.  ‘Die  most  important  of  these  actioiu  are 
flying  thrmgh  the  world,  scaling  t^  world,  and  grabbing 
objects.  This  paper  shows  how  these  actkms  can  be  precisely 
specified  with  firame-to-frame  invariants,  and  how  die  code  to 
implemam  the  actions  can  be  darived  from  the  mvariants  by 
algebraic  manipiilation. 

INTRODUCTION 

Wearing  a  Hoad-Mounted  Display  (HMD)  gives  a  human  user 
the  setuadon  of  being  insida  a  tfarae-dirnansional  oontpoter- 
simulated  world.  Baamse  the  HNU)  replaces  the  sights  and 
sounds  of  tha  teal  world  with  a  computer-gmeratad  virtual 
world,  this  synthasizad  world  is  called  vir^  reality. 

*11)0  virtual  world  siorounding  the  user  is  defined  by  a  graphics 
datibase  called  a  modal,  which  gives  tbs  colon  and  coordinates 
for  each  of  the  polygons  making  up  the  virtual  worid.  The 
polygons  making  up  the  virtual  world  arc  normally  grouped 
into  antitias  called  objtets,  cadi  of  which  hat  its  own  location 
and  otiouaiiaa.  The  human  being  wearing  As  HMD  is  called 
the  asar.  and  also  has  a  location  and  oricntatioa  within  the 
virtual  world. 

To  nira  the  data  in  the  model  into  the  iSasion  of  a  surrounding 
virtual  world,  the  HMD  system  requires  hardware 
components.  The  tracker  measures  the  position  and 
orientation  of  the  usm't  head  and  hwvt  The  graphics  engine 
generates  the  images  seat  by  the  user,  which  ate  then  displayed 
on  the  HMD.  Tbs  ma/tmai  input  device  allows  the  usa  to  use 
gesnaes  of  the  hand  to  cause  thinp  to  happen  in  the  vitttal 
world. 

BASIC  ACTIONS 

An  action  changa  the  state  of  the  virtual  world  or  the  uta's 
viewpoim  within  it  nnda  control  of  a  gature  of  the  hand,  as 
meastaed  by  the  manual  input  device.  The  hand  gesture 
initiates  and  terminates  the  action,  end  Ae  changing  position 


and  orientation  of  the  hand  during  the  gesture  is  also  used  to 
control  what  happens  u  Ae  action  progresses. 

The  manual  input  device  may  be  a  hand-held  manipulandum 
wiA  pushbuttons  on  it,  or  it  may  be  an  mstrumented  glove.  In 
eiAa  case,  the  position  and  orietuation  of  the  input  device 
most  be  measured  by  the  tracker  to  enable  msnual  control  of 
actions.  The  inpat  device  must  also  allow  the  user  to  signal  to 
the  system  to  start  and  stop  actions,  attd  to  select  among 
altenative  actions. 

Certain  fundamental  manually-controlled  actioiu  msy  be 
implemented  for  any  vimial  world.  These  actioiu  mvoWe 
changing  the  loeati^  orientation  or  scale  of  either  an  object 
or  a  user,  as  shown  in  Table  1. 


Trarulate 

Rotate 

Scale 


Usa 

Object 

fly  through 

grab  (sod  move)  object 

Ae  world 

tilt  Ae  worid 

grab  (and  turn)  object 

expand  or  shrink 

scale  object 

Ae  world 

Tdile  1.  Buie  actioiu 


Flying  is  defined  here  u  an  operation  of  translating  in  Ae 
directiop  pointed  by  the  hand-held  irqwt  device,  wiA  steering 
dons  by  efaanpr^  Ae  hand  orientation.  This  is  different  from 
the  type  of  flying  availabie  m  a  flight  simnlaior,  where  the  user 
can  not  only  translate  but  can  also  cause  Ae  virtual  world  to 
rotate  asou^  him  by  banking.  However,  translation-only 
flying  is  appropriate  for  a  HMD  because  the  user  hu  Ae  ability 
to  turn  and  look  in  any  direction,  and  to  poim  Ae  input  device 
in  any  diraction.  We  believe  that  keepmg  the  orientation  of 
the  virtual  arorld  locked  to  that  of  Ae  real  arorld  helps  Ae  user 
10  navigate  arhile  flying  through  Ae  virtual  world. 

Tilling  the  world  is  Ae  ability  lo  re-orient  Ae  virtual  world 
relative  lo  Ae  usa't  orientation;  that  is,  to  turn  Ae 
surrounding  virtual  world  sideways.  This  is  implemented  by 
rotating  Ae  user  ariA  respea  to  Ae  virtual  srorld.  which  is 
subjectivdy  perceived  by  Ae  user  u  the  entire  virtual  world 
rotating  around  him. 

Sealing  the  world  is  Ae  capability  to  shrink  or  expand  Ae 
world  relative  to  Ae  user,  u  occurs  to  Alice  m  Wonderland 
when  she  drinks  fiom  Ae  hole  bottle  or  eats  Ae  little  cake.  By 
setting  up  the  action  code  properly,  the  user  can  shrink  and 
expand  the  world  while  manually  steering  Ae  center  of 


1 


cxptntkm.  This  saabiw  a  powerful  mediod  of  nvel  in  very 
large  virtual  worlds:  the  user  shrinks  the  world  down  until  the 
destination  is  within  arm’s  reach  and  then  expands  the  world, 
continuously  steering  the  center  of  expansion  so  as  to  arrive  at 
the  correctly-scaled  destination. 

Crabbing  an  object  is  picking  up  and  moving  a  simulated 
object  that  appears  in  the  virtual  world.  By  analogy  with  real- 
world  grabbing  of  objecu,  this  includes  the  ability  to  rotate 
the  held  object  before  releasing  it 

Scaling  an  object  is  just  shrinking  or  expanding  an  individual 
object  alone. 

This  paper  seeks  to  answer  die  following  questaon:  How  can  die 
basic  actions  of  flying,  grabbing,  scaling  and  tilting  in  a  HMD 
system  be  specified  and  implemgiled? 

PRIOR  WORK 

The  flrst  HMD  was  built  in  1968  by  Ivan  Sutherland  [8],  but 
since  it  had  no  mmual  input  device  other  than  a  keyboard,  it  did 
not  allow  actions  controlled  by  manual  gestures.  At  the 
University  of  Utah,  a  tracked  manual  input  device  called  a 
"wand”  was  added  to  the  system  [9].  TIm  t^  of  the  wand  wu 
tracked  in  position  but  not  orientation.  *nve  wand  was  used  to 
deform  the  surfaces  of  virtual  objects  composed  of  curved 
patches  [2]. 

In  198S  at  NASA  Ames  Research  Center,  McGreevy  and 
Humphries  built  a  HMD  ivhich  was  later  improved  by  Rsher, 
Robineti  and  others  [3]-  Under  comract  lo  NASA,  VPL 
Research  provided  an  nutrumented  glove,  later  named  the 
'DataGlove,'  which  served  as  a  manual  input  device.  The 
position  of  the  hand  and  head  were  tracked  with  a  PoDiemus 
3Spaca  magnetic  tracker.  In  1986  using  the  glove  irqiut 
device,  Robines  implemented  on  diis  system  the  acdons  of 
flying  through  the  worid,  scaling  the  world,  rotating  the  amrld, 
arid  grabbing  objects. 

Some  of  these  actions,  particularly  flying  and  grabbing 
objects,  have  since  ueen  implemented  on  HMD  systems  at 
several  sites.  VPL  Research  began  in  1989  selling 
commercially  a  HMD  system  that  used  a  glove  to  control  the 
actions  of  flying  and  grabbing  [1].  At  the  University  of  North 
Caroliru  [7]15],  the  actkmt  of  flying,  scaling  and  grabbing 
were  controlled  adth  a  hand-held  manual  irgmt  device  with 
pushbuttons  on  it  which  was  made  from  a  billiard  balL 

COORDINATE  SYSTEMS  DIAGRAM  FOR  A  HMD 

Various  coordinate  systems  co-ex;  r  within  a  HMD  system.  All 
of  these  coordinate  systems  exist  simaltaneously,  and  although 
over  time  they  may  be  moving  with  respect  to  one  ■lothm,  at 
any  given  moment  each  pair  of  them  has  a  relative  position  and 
orientation.  The  instantaneous  relationship  between  two 
coordinate  systems  can  be  described  widi  a  traruform  that 
converts  the  coordinates  of  a  point  described  in  one  coordinate 
system  to  the  coordinates  that  represent  that  same  point  in  the 
second  coordinate  system. 

Although  transforms  exist  between  any  pair  of  coordinate 
systems  in  the  HMD  system,  certain  pairs  of  coordinate 
systems  have  relative  positions  that  are  either  constant, 
measured  by  the  tracker,  or  are  known  for  some  other  reason. 
These  are  the  independent  transforms,  which  are  shown  in 
relation  to  one  another  in  Figure  1.  In  this  diagram,  each  node 
stands  for  a  coordinate  system,  and  each  edge  linking  two 


nodes  stands  for  a  transform  between  those  two  coordinate 
systems. 

modified  when  user  flies,  modified  when 


Rgure  1.  Cooidinaie  systems  diagram  for  a  single-user 
HMD  system 

NOMENCLATURE  FOR  TRANSFORMS 

We  abbreviate  the  cooidinate  systems  with  the  first  letters  of 
their  names,  The  World-Objeo  transform  may  be  written  as 
Two-  Transform  Two  converts  a  point  Pq  in  cooidinate  system 
O  to  a  point  Pw  in  coordinate  system  W. 

Fw  ■  Two '  Fo 

This  notation  is  similar  to  that  used  in  [4].  Notice  that  the 
subacripts  cancel  nioely,  as  in  [6].  Likewise,  the  composition 
of  the  transform  Two  fciog  from  O  to  W  with  the  transform 
Tgw  going  from  W  to  R  gives  a  transform  Tkq  from  O  to  R, 
with  die  cancellation  rule  working  here,  too: 

TgwTwo*T*o 

The  invene  of  transform  Two  >>  ^rinen  Tqw- 

SPECIFYING  ACTIONS  WITH  INVARIANTS 

An  action  in  a  virtual  world  is  performed  by  activating  the 
input  device,  such  u  by  {wshing  a  button,  and  then  moving  the 
input  device  to  control  the  action  as  it  progresses.  As  an 
example,  grabbing  a  simulated  object  requires,  for  each  frame 
while  the  grab  action  is  in  progress,  that  a  new  position  for  the 
object  be  computed  based  on  the  changing  position  of  the 
user's  hand. 

It  is  possible  to  precisely  define  grabbing  and  other  actioru 
with  an  invariant,  which  is  an  equation  that  describes  the 


desired  relaiionthip  emong  certain  tnnsfonns  involved  in  the 
action.  The  invahant  is  typically  stated  as  a  relation  between 
certain  transforms  in  the  current  display  frame  and  certain 
transforrru  in  the  previous  frame.  In  the  case  of  grabbing,  the 
invariant  to  be  maintained  is  that  the  Object-Hand  transform  be 
equal  to  iu  value  in  the  previous  frame  while  the  grab  action  is 
in  progress;  in  other  words,  that  the  object  remain  frxed  with 
respect  to  the  hand  while  it  is  being  gnJibed. 

Starting  from  the  invariant  and  a  diagram  of  the  coordinate 
systems  invoNed,  a  mathematical  derivation  can  be  performed 
which  prodacea  a  fanmila  far  updating  the  proper  transform  to 
cause  ^  desired  eetkm  to  occur.  For  grabbing,  this  would  be 
"P*»r»"g  the  Object-World  transform  to  change  the  objea’s 
position  and  orientation  in  the  vittaal  worid. 

Rigorously  dariviag  the  update  formula  from  a  sitn|de  invariant 
is  much  easier  and  more  reliable  than  attempting  to  write  down 
the  update  formula  using  the  coordinate  systems  diagram  and 
informal  reasoning.  Also,  the  matching  of  adjacent  subacripu 
in  the  notation  helps  to  check  that  the  transfontu  ere  in  correct 
order. 

GRABBING  AN  OBJECT 

To  derive  the  update  formula  for  grabbing,  we  first  look  at  dM 
relevant  part  of  the  coordinate  system  diagram,  shown  in 
Figure  2. 


held  fixed 
during  grab 


modsTiedby 
grab  action 


tracker 


N  ■ '  \ 

I 

measured  by  is 

itadeer  on 


I  for  the  grab  actioa  the  invariant 

measured  by  is  to  keep  this  single  transform 

Itadeer  unchanged  from  frame  to  frame 

Figure  2.  Coordiiuie  systems  diagram  for  grabbing  an 
object 

A  way  of  describing  the  action  of  grabbing  is  that  the  Object- 
Hand  transform  Tqh  remain  unchanged  from  firama  to  frai^ 
which  is  expressed  by  the  invariam 

Toh’  •  Toh 

where  the  apostrophe  in  Tqh'  indicates  a  traruform  in  the 
current  frame  which  is  being  updated,  and  no  apostrophe  means 
the  value  of  the  transform  from  the  previous  frame. 


To  move  an  individual  object,  the  Object-World  transform  Tq^ 
must  be  updated  each  frame  in  a  way  that  pneserves  the 
invariant.  To  derive  the  uptdate  formula  for  grabbing,  wc  start 
with  the  invariant  and  decompose  the  transforms  on  both  sides 
based  on  the  reIationshi]»  among  the  coordinate  systems  as 
shown  in  the  coordinate  system  diagram. 

Tow*  •  Twr*  ■  Trx'  •  Tto’  •  Tow  •  Twr-  Tux  •  Txh 

Wc  then  use  algebraic  manipiulatioru  to  isolate  the  desired 
transform  on  the  left  side  of  the  equation,  remembering  that 
these  transforms  are  not  commutative. 

Tow*  Tw»*  T„*  -  Tow  •  Twit  •  Tkt  •  Tto  Tot' 

Tow’-Twr*  ■  TowTanfTjx-TxH'Tin-’ -Tjh* 

Tow’  ■  Tow-Twa-Tgr-TTH-Tur’ -Ttr’  Titw’ 

Tliit  it  the  update  forrtuila  for  grabbing,  which  updates  the 
Object-World  traruform  baaed  on  is  previous  value,  the  current 
and  previoni  vahiea  of  the  Head-Tracker  traruform  (which 
diangea  at  the  hand  tttovea),  attd  the  values  of  the  intervening 
ttaruforms  between  Tracker  and  WoHd.  The  effect  of  executing 
thia  asignment  each  frame  is  to  keep  the  object  in  a  fixed 
potidon  and  orientation  relative  to  the  hand,  even  though  the 
hand  is  moving  around  within  the  virtual  world. 

Another  action  which  can  be  implemeiued  in  a  similar  manner 
is  “grabbing  the  fabric  of  space.“  In  this  esse,  the  user  can 
grab  asd  dll  the  entire  virtual  world,  rather  than  just  s  single 
object,  by  holding  the  World-Hand  traruform  invariam  while 
the  lumd  rotates. 

FLYING 

The  action  of  flying  is  trantleting  the  user  through  the  virtual 
worid  in  the  direction  pointed  by  the  manual  input  device.  The 
user  Steen  by  routting  the  menual  input  device  as  the  flight 
proceeds.  A  inetqphor  for  this  rypie  of  flying  is  that  the  user 
holds  e  rocket  pistol  in  his  hand,  which  drags  him  through  the 
virual  world  when  he  squeezes  the  trigger. 

The  manual  inpat  device  is  considered  to  poim  in  a  particular 
direction  that  ii  relative  to  iu  local  coordinate  system.  This 
may  be  thought  of  w  a  3D  vector  in  Hand  coordinates,  where 
the  vector’s  length  specifies  the  flying  ^leed  aid  the  vector’s 
direction  defines  the  direction  the  input  device  points.  This 
vector  defines  a  translation  transform,  '^nitninitl  -  whidi 
moves  a  point  in  Hand  coordinates  to  a  new  pxuition  in  Hand 
coordinates.  To  inqtlemem  flying,  we  first  need  to  conven  this 
traufoimetion  to  operate  on  poinu  in  Room  coordinates. 


’  ■  TgH*  Thb,.,^’  Thr’ 


To  make  the  user’s  jtosition  change  within  the  virtual  world, 
die  World-Room  traruform  must  be  modified  each  frame,  so  the 
ioveiiam  for  Oying  is 

Tw»’  -Tw*  •  TrmmImr’ 

which  may  be  expianded  to  give  the  upxlaie  formula  for  flying. 


Twr’  ■  Twr  ■  Trx’  •  Txh’  •  THmaUuH  •  Tht’  •  Txr’ 

SCALING  THE  WORLD 

It  it  pwssible  to  shrink  or  expand  the  surrounding  virtual  world. 
This  is  compreheruible  and  effective  because  the  user  has  direct 


perception  of  the  lizc  of  aid  distance  to  vinual  objects  through 
stereopsis  and  head-motion  parallax,  and  can  therefore  easily 
perceive  the  concerted  motions  of  the  objects  in  the  vimial 
world  expanding  around  a  center  of  expansion,  or  shrinking 
towards  a  center  of  contraction. 

The  type  of  scaling  used  is  uniform  scaling,  in  which  all  three 
dimensions  are  always  scaled  by  the  same  factor.  There  is 
always  a  center  of  scaling  when  uniform  scaling  occurs,  and  for 
the  manually  controlled  action  of  scaling  die  worid,  it  makes 
sense  to  kwau  the  center  of  acaling  at  the  uaer's  hand.  When 
expanding  the  world,  the  center  of  scaling  is  the  point  that 
virtual  objects  move  awqr  fiom  u  expansion  occurs,  and  so  to 
end  up  at  a  specific  desired  location  within  a  fotmcrly-tiny 
virtual  world,  the  center  of  scaling  must  be  repeatedly  re- 
centered  on  the  desired  location  as  it  emerges  dtoing 
expansion. 

Implementing  this  action  requires  a  dcrivadon  similar  to  that 
us^  for  flying.  An  incraraental  sealing  transformation  in  Hand 
coordinates,Tibc^,  win  use  the  Hand  origin  as  the  center  of 
scaling.  Below  we  give  the  invmam  for  sealing  the  world,  and 
the  update  formula  derived  horn  it 

Twr’  ■  ■  flMSklt’ 

Twr’  *  Twr  •  Tgt'  •  Tth*  •  Ti^awi  •  Tht’  •  Tt»' 

GENERAL  FORM 

Upon  examining  die  mvariants  for  flying  and  scaling,  we  see  a 
strong  similarity  between  them:  both  mvariants  are  of  the 
form: 

Twr'  •  Twr  ■  Tg  -i— fimR' 

In  (act,  these  tsro  utvarianti  for  updating  Twr  ■«  examples  of  a 
more  general  technique  for  updating  a  uinsform  between  two 
coordinate  tystenu  based  on  a  transform  that  occurs  in  a  third 
coordinate  systeriL  The  general  form  for  updating  the 
transform  T^r  in  temu  of  an  action  in  coordinate  system  K  is: 

Tab'  ■  Tab  *  Tbr'  ■  TK«Bi^bav< '  Td' 

where  there  may  be  an  arbitrary  number  of  ooordinase  systems 
between  B  and  K.  nd  Tgg  is  the  product  of  the  transforms  that 
go  between  the  two  coordinate  systams. 

Using  this  general  form,  acaling  an  object  about  the  is 
analogous  to  acaling  the  world  about  the  hand: 

Tow'  ■  Tow  •  Twh'  •  TumM  •  Thw' 

CONCLUSIONS 

The  foregoing  examples  of  grabbing,  flying  and  scaling  show 
how  acdotu  can  be  implemented  that  operate  under  contitnious 
mmual  control  by  the  user.  For  each  action,  the  relationship 
between  the  motion  of  the  hwid  md  the  transforms  to  be 
modified  was  precisely  specified  with  an  invariant.  These 
invariants  not  only  provided  a  concise  and  precise 
speciHcaiion  of  each  action,  but  also  provided  a  starting  point 
for  a  formal  derivation  that  produced  update  equations  which 
could  be  used  directly  to  implement  the  actxms. 

Using  invariants  and  derivations  to  produce  the  code  to 
implement  grabbing,  scaling  and  flying  is  greatly  superior  to 
the  method  which  is  often  used,  namely,  to  just  write  down  a 


sequence  of  transforms  that  looks  right  based  on  the  coordinate 
system  diagram.  It  is  easy  to  get  tome  of  the  transforms  in  the 
wrong  order.  The  notation  used  in  this  paper  provides  a  check 
against  misordering  the  trarufomu  by  requiring  adjacent 
sttbacripts  to  match  The  HMD  software  at  UNC  was 
implernented  using  this  notation  and  the  formulas  derived  in 
this  paper,  and  serves  at  proof  that  they  work. 
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Abttrsct 

An  optoelecaonic  head-tracking  system  for  head-mounted 
displays  is  described.  The  system  features  a  scalable  work  area 
that  currently  measures  10'  x  12'.  a  measurement  update  rate  of 
20-100  Hz  with  20-60  ms  of  delay,  and  a  resolution 
specification  of  2  mm  and  0.2  degrees.  ‘Hie  sensors  consist  of 
four  head-mounted  imaging  devices  that  view  infrared  light- 
emitting  diodes  (LEDs)  mounted  in  a  10'  x  12'  grid  of  modular  2' 
X  2'  suspended  ceiling  panels.  Photogrammetric  techniques 
allow  the  head's  location  to  be  expressed  as  a  function  of  the 
known  LED  positions  and  their  projected  images  on  the 
sensors.  The  work  area  is  scaled  ^  simply  adding  panels  to 
the  ceiling's  grid.  Discontinuities  that  occur^  when  changing 
working  sets  of  LEDs  were  reduced  by  carefully  managing  all 
error  sources,  including  LED  placement  tolerances,  and  by 
adopting  an  overdetermined  mathematical  model  for  the 
computation  of  head  position:  space  resection  by  collinearity. 
The  working  system  wu  demonstrated  in  the  Tomorrow's 
Realities  gallery  at  the  ACM  SIGGRAPH  '91  conference. 

CR  categories  and  subject  descriptors:  1.3.1 
(Computer  Graphics]:  Hardware  Architecture  -  three- 
dimensional  displays',  13.7  (Computer  Graphics]:  Three- 
Dimensional  Grai^ics  and  Realu  .  -  Virtual  Reality 

Additional  Key  Words  and  Phrases:  Head-mounted 
displays,  head  tracking 


1  Introduction 

It  is  generally  accepted  that  deficiencies  in  accuracy, 
resolution,  update  rate,  and  lag  in  the  measurement  of  head 
position  can  Aversely  affect  the  overall  performance  of  a  H,MD 
il7]{24](25].  Our  experience  suggests  that  an  additional 
specification  requires  more  emphasis:  range. 
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Figure  1;  The  existing  system  in  UNO's  graphics  laboratory 


Most  existing  HMD  trackers  were  built  to  support  situations 
shat  do  not  require  long-range  tracking,  such  as  cockpit-like 
environments  where  the  user  is  confined  to  a  seat  and  the  range 
of  head  motion  is  limited.  But  many  virtual  worlds 
applications,  such  as  architectural  walkthroughs,  would  benefit 
from  more  freedom  of  movement  (Figure  2).  Long-range 
trackers  would  allow  greater  areas  to  be  explored  naturally,  on 
foot,  reducing  the  need  to  resort  to  techniques  such  as  flying  or 
walking  on  treadmills. 

Such  techniques  of  extending  range  work  adequately  with 
closed-view  HMDs  that  completely  obscure  reality.  With  see- 
through  HMDs  (9][11],  however,  the  user's  visual  connection 
with  reality  is  intact  and  hybrid  applications  are  possible 
where  physical  objects  and  computer-generated  images  coexist. 
In  this  situation,  flying  though  the  model  is  meaningless.  The 
model  is  registered  to  the  physical  work!  and  one's  relationship 
to  both  must  change  simultaneously. 

This  paper  describes  the  second  generation  of  an 
optoelectronic  head-tracking  concept  developed  at  the 
University  of  North  Carolina  at  Chapel  Hill.  In  the  concept  s 
first  generation,  the  fundamental  design  parameters  were 
explored  and  a  bcnch-iop  prototype  was  constructed  [2K| 
Building  on  this  success,  the  sccond-gcncration  tracker  is  a 


fully  funciiontl  prototype  that  significantly  extends  the 
xraritspace  of  an  HMD  wearer. 


Figurt  2:  Walkthrough  of  Brooks'  kitchen  design  that  runs  with 
the  tracker.  Actual  resolution  of  images  seen  in  the  HMD  is 
much  tower  than  this  picture's  resolution. 


The  cunent  lystem  (Figure  1)  pieces  four  outwar<l>k>oking 
image  senson  on  the  wearer's  head  and  locates  LEDs  in  a  10'  a 
12'  suspended  catling  structure  of  modular  2'  x  2*  ceiling 
panels.  Each  pand  houses  32  LEDs,  for  a  total  of  960  LEDs  in 
the  ceiling.  Images  of  LEDs  are  formed  by  lateral>effect 
photodiode  detectors  within  each  head>mount^  sensor.  The 
location  of  each  LED's  image  on  a  detector,  or 
photocoordinate,  is  used  along  with  the  known  LED  locations 
in  the  ceiling  to  compute  the  head's  position  and  orientation. 
To  enhance  resolution,  the  field  of  view  of  each  sensor  is 
narrow.  Thus,  as  shown  in  Rgures  3  and  7,  each  sensor  sees 
only  a  small  number  of  LEDs  at  any  iiutanL  As  the  user  moves 
about,  the  working  set  of  visible  LEDs  changes,  making  this  a 
cellular  head-tracking  system. 

Measurements  of  head  position  and  orienuiion  are  produced  at 
a  rate  of  20-100  Hz  with  20-60  ms  of  delay.  The  system's 
accuracy  has  not  been  measured  precisely,  but  the  resolution  is 
2  mm  and  0.2  degrees.  It  was  demoiutrated  in  the  Tomorrow's 
Realities  gallery  at  the  ACM  SIGGRAPH  '91  conference,  and  is, 
to  our  knowledge,  the  Hist  demoiutrated  scalable  head-tracking 
system  for  HMDs. 

The  system  is  novel  for  two  reasons.  First,  the  sensor 
conFiguration  is  unique.  Other  optical  tracking  systems  fix  the 
scTuors  in  the  environment  aiKf  mount  the  LEDs  on  the  moving 
body  [30].  The  outward-looking  configuration  is  superior  for  it 
improves  the  system's  ability  to  detect  head  rotaticn.  The 
scalable  work  space  is  the  system's  second  contribution.  If  a 
larger  work  space  is  desired,  more  panels  can  be  easily  added  lo 
the  overhead  grid. 


2  Previou.s  work 

Many  tracking  systems  precede  this  efforu  and  we  will  briefly 
survey  representative  examples.  The  essence  of  the  problem  is 
the  realtime  measurement  of  the  position  and  orientation  of  a 
rigid  moving  body  with  respect  to  an  absolute  reference  frame, 
a  six-degrec-of-frccdom  (6DOF)  measurement  problem. 
Solutions  arc  relevant  to  many  other  fields. 

To  our  knowledge,  four  fundamentally  different  technologies 
have  been  used  to  track  HMDs:  mechanical,  magnetic, 
ultrasonic,  and  optical. 

The  first  HMD.  built  by  Ivan  Sutherland  [27],  used  a  mechanical 
liitkage  to  measure  head  position.  A  commercial  product.  The 
Boom  [12],  uses  a  mechanical  linkage  to  measure  the  gaze 
direction  of  a  hand-held  binocular  display.  The  Air  Force 
Human  Resources  Laboratory  (AFHRL)  uses  a  mechanical 
linkage  to  measure  the  position  and  orienution  of  a  HMD  used 
for  simulation  [24],  Mechanical  systems  have  sufficient 
accuracy,  resolution,  and  frequency  response,  yet  their  range  is 
severely  limited,  and  a  mechanical  tether  is  undesirable  for 
many  applications. 

Magnetic-based  systems  [3][2t]  are  the  most  widely  used  hand 
and  head  trackers  today.  They  are  small,  relatively 
inexpensive,  and  do  not  have  line-of-sight  restrictions.  Their 
primary  limitations  are  distortions  caused  by  metal  or 
electromagnetic  fields,  and  limited  range  [13]. 

Ultrasonic  approaches  have  also  been  successful,  such  as  the 
commercially-available  Logitech  tracker  [20].  Time-of-flight 
measurements  are  used  to  triangulate  the  positions  of  sensors 
mounted  on  the  HMD.  The  strength  of  this  technology  is 
minimum  helmet  weight  [13).  Physical  obscuration  as  well  as 
reflections  and  variations  of  the  speed  of  sound  due  to  changes 
in  the  ambient  air  density  make  it  difficult  to  maintain  accuracy 
[5]. 

Because  of  the  potential  for  operation  over  greater  distances, 
optical  approaches  are  plentiful,  and  it  is  helpful  to  categorize 
them  on  the  basis  of  the  light  source  used.  Visible,  infrared, 
and  laser  light  sources  have  each  been  exploited. 

Ferrin  [13]  reports  the  existence  of  a  prototype  helmet  tracking 
system  using  visible  light.  Although  it  only  tracks 
orienution,  it  is  worth  mentioning  here  because  of  its  unique 
approach.  A  patterned  target  is  placed  on  the  helmet  and  a 
cockpit-mount^  video  camera  acquires  images  in  real  time.  The 
pattern  is  designed  to  produce  a  unique  image  for  any  possible 
head  orientation.  The  strength  of  this  approach  is  the  use  of 
passive  targets  which  minimize  helmet  weight.  Reflections 
and  other  light  sources  are  potential  sources  of  error. 

Bishop's  Self-Tracker  [7]  is  a  research  effort  involving  visible 
light  A  Self-Tracker  chip  senses  incremental  displacements 
and  rotations  by  imaging  an  unstructured  scene.  A  head- 
mounted  cluster  of  these  chips  provide  sufficient  information 
for  the  computation  of  head  position  and  orientation. 
Although  still  under  development,  the  concept  is  mentioned 
here  because  it  would  allow  an  optical  tracking  system  to 
operate  outdoors,  where  a  structured  environment,  such  as  our 
ceiling  of  LEDs,  would  be  impossible  to  realize. 
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Because  of  ihe  difficulties  associated  with  processing 
information  in  an  unstructured  scene,  most  high-speed  optical 
measurement  systems  use  highly-structured  infrared  or  laser 
light  sources  in  conjunction  with  solid-state  sensors.  The 
sensor  is  a  often  a  lateral-effect  photodiode  as  opposed  to  a  true 
imaging  device,  because  the  photodiode  produces  currents  that 
are  directly  related  to  the  location  of  a  light  spot's  centroid  on 
iu  sensitive  surface  [32].  The  resultant  sensor  is  relatively 
insensitive  to  focus,  and  the  light  spot's  location,  or 
photocoordinate,  is  immediately  available  without  the  need  for 
image  processing. 

During  the  19'70's,  Selspot  [23]  popularized  the  use  of  infrared 
LEDs  as  targets  and  lateral-effect  photodiodes  as  sensors  in  a 
commercially-available  system.  Their  primary  emphasis  wu. 
and  still  is.  on  the  three-dimensional  locations  of  individual 
targeu.  That  is,  the  Selspot  system  does  not  automate  the 
computation  of  a  rigid  body's  orientation.  In  a  response  to  this 
shortcoming.  Antonsson  [2]  refined  the  Selspot  system  for  use 
in  dynamic  measuremenu  of  mechanical  systems.  The 
resultant  system  uses  two  Selspot  cameras  to  view  a  moving 
body  instrumented  with  LEDs.  Similar  approaches  have  been 
applied  to  HMD  systems  in  cockpits  [13]  and  in  simulators 
[11]. 

The  use  of  an  LED  light  source  limits  the  range  of  these 
systems.  Typically,  the  distance  between  source  and  detector 
can  be  no  greater  than  several  feet.  Longer  distances  can  be 
spanned  with  laser  light  sources. 

The  only  known  example  of  a  6DOF  tracker  using  laser  sources 
is  the  Minnesou  Scanner  [26].  With  this  sytem.  scanning 
mirrors  are  used  to  sweep  orthogonal  stripes  of  light  across  the 
working  volume.  Photodiodes  are  both  ftxed  in  space  and 
placed  on  the  moving  body.  By  measuring  the  time  tetween  a 
light  stripe's  contact  with  a  fixed  and  moving  photodiode,  the 
diode's  three-dimensional  location  can  be  computed.  Given  the 
location  of  three  or  more  moving  diodes,  the  moving  body's 
orientation  can  be  computed.  Similar  technology  has  been 
applied  to  the  cockpit,  although  orienution  was  the  only 
concern  [13]. 


Figure  J  Conceptual  drawing  of  outward- looking  ay  stem  and 
the  sensors'  fields  of  view 


3 

Wang  dcnuinstraicd  the  viability  of  hcad-mounicd  Ijicral-cffcci 
photodiodes  and  overhead  LEDs.  This  system  extends  his  work 
in  several  ways.  First,  an  overhead  grid  of  960  LEDs  was 
produced  with  well-controlled  LED  location  tolerances,  and 
more  attention  W3.s  paid  to  controlling  other  error  sources  as 
well.  Second,  mathematical  techniques  were  developed  that 
allow  an  arbitrary  number  of  sensors  and  an  arbitrary  number  of 
LEDs  in  the  field  of  view  of  each  sensor  to  be  used  in  the 
computation  of  head  location.  This  resulted  in  an 
overdetermined  system  of  equations  which,  when  solved,  was 
less  susceptible  to  system  error  sources  than  the  previous 
mathematical  approach  [10].  Third,  the  analog  signals 
emerging  from  the  sensors  were  digitally  processed  to  reject 
ambient  light.  Finally,  techniques  for  quickly  determining  the 
working  sets  of  LEDs  were  developed. 

3,1  Sensor  configuration 

Typically,  optical  trackers  are  inward-looking;  sensors  arc 
fixed  in  the  environment  within  which  the  HMD  wearer  moves. 
With  Self-Tracker.  Bishop  and  Fuchs  introduced  the  concept  of 
outward- looking  trackers  that  mount  the  image  sensors  on  the 
head,  looking  out  at  the  environment  (Figure  3). 

If  a  large  work  area  is  required,  outward-looking  configurations 
have  an  advantage  over  inward-looking  techniques  when 
recovering  orientation.  The  two  are  equivalent  for  measuring 
translation;  moving  the  sensor  causes  the  same  image  shift  as 
moving  the  scene.  Rotations  are  significantly  different. 
Unless  targets  are  mounted  on  antlers,  an  inward-looking 
sensor  perceives  a  small  image  shift  when  the  user  performs  a 
small  head  rouiion.  'The  same  head  rotation  creates  a  much 
larger  image  shift  with  a  head-mounted  sensor.  For  a  given 
sensor  resolution,  an  outward- looking  system  is  more 
sensitive  to  orienution  changes. 


Figure  4:  Remote  Processor  and  head  unit  with  four  sensors 


To  improve  resolution  in  general,  long  focal  lengths  must  be 
used  with  ail  optical  sensor  regardless  of  whether  the 
configuration  is  inward  or  outward-looking.  Thus,  a  wide-angle 
lens  cannot  significantly  extend  the  work  area  of  an  inward¬ 
looking  system  without  sacrificing  resolution  and  accuracy. 

Narrow  fields  of  view  arc  a  consequence  of  long  focal  lengths 
Therefore,  the  HMD  wearer  cannot  move  very  far  before  an  LED 
leaves  a  given  scn.sor  s  field  of  view.  One  solution  is  a  cellular 


army  of  eiiher  LFDs  or  dcioctors.  For  an  infrared  system  using 
LEDs  and  lateral-cfTcct  photodiodes,  system  cost  is  minimized 
by  replicating  LEDs  as  opposed  to  sensors.  This  is  a  result  of 
both  the  device  cost  as  well  as  the  required  support  circuitry. 

In  the  current  system,  four  Hamamatsu  (model  SI 880)  sensors 
are  mounted  atop  the  head,  u  shown  in  Figure  4.  Each  sensor 
eonsisu  of  a  camera  body  to  which  a  Fujinon  lens  (model  CF 
SOB)  is  attached.  The  focal  length  of  each  letu  is  SOmm.  Their 
principal  poinu  were  determined  experimentally  by  an  optical 
laboratory.  An  infrared  filter  (Tiffen  87)  is  used  to  reject 
ambient  light. 

3.2  Beacon  configuration 

Experience  with  simulations  and  an  early  48-LED  prototype 
revealed  the  problem  of  beacon  swiicking  error:  as  the  user 
moved  around  and  the  working  set  of  beacons  changed, 
discontinuous  jumps  in  position  and  orientation  occurred. 
These  are  caused  by  errors  in  the  sensor  locations,  disttirtions 
caused  by  the  lens  md  photodiode  detector,  and  errors  in  the 
positions  of  the  beacons  in  the  ceiling. 

To  control  beacon  locations,  we  housed  the  LEDs  in  carefully 
constructed  ceiling  panels.  Each  2'  x  2'  panel  is  an  anodized 
aluminum  enclosurt  that  encases  a  20*  x  20*  t«vo-sided  primed 
circuit  board.  On  this  board  are  electronics  to  drive  32  LEDs. 
The  LEDs  are  mounted  in  the  front  surface  with  standard  plastic 
insets.  Using  standard  electronic  enclosure  manufacturing 
techniques,  it  was  relatively  easy  to  realize  an  LED-to-LED 
centerline  spacing  tolerance  of  .003*  on  a  given  panel. 

The  panels  are  hung  from  a  Unistrut  supentructure  (Figure  1). 
At  each  interior  verux  of  a  2‘  x  2'  grid,  a  vertically  adjusuble 
hanger  mates  with  four  panels.  Four  holes  in  the  face  of  a  panel 
slide  onto  one  of  four  dowels  on  each  hanger.  The  entire  array 
of  panels  is  levelled  with  a  Spectra  Physics  Laser-Level,  which 
establishes  a  plane  of  visible  red  light  several  inches  below  the 
panels'  faces.  Each  hanger  it  designed  to  accept  a  tensor 
(Induttra-Eye)  that  measures  the  vertical  position  of  the  laser 
relative  to  its  own  case.  By  moving  the  hangen  up  or  down, 
they  can  be  aligned  to  within  .006*  of  the  light  beam. 

The  panels  are  electricaOy  connected  by  a  dau  and  power  daisy 
chain.  The  data  daisy  ^ain  allows  an  individual  LED  to  be 
selected.  Once  selecter*,  the  LED  (Siemens  SFH  487P)  can  be 
driven  with  a  programmable  cunent  that  rangu  from  0-2 
amperes.  The  programmable  current  allows  an  electronic  iris 
feature  to  be  implemented.  Typically,  an  LED  will  be  on  for  no 
more  than  200  )tsec.  During  this  time  period,  the  cunent  is 
adjusted  to  achieve  a  desired  signal  level  at  the  sensor  (see 
Section  4). 


The  LED  Manager  is  a  68030-ba5ed  processing  module  that 
controls  the  Remote  Processor  as  well  as  the  ceiling.  A  TAXl- 
based  serial  datalink  [1]  provides  access  to  the  Remote 
Processor  while  the  ceiling's  data  daisy  chain  terminates  at  the 
LED  Manager.  Software  executing  on  this  module  is 
responsible  for  turning  LEDs  on  and  for  extracting  data  from 
the  sensors.  The  LED  Manager  resides  in  a  remote  VME  chassis 
that  must  be  located  near  the  ceiling  structure. 
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LED  Manager  Queue  Manager  Collinearity 
Figure  5:  System  Dataflow 


For  each  measurement  of  head  location,  the  LED  Manager 
produces  a  list  of  visible  LEDs  and  their  usociated 
photocoordinates.  'Hus  list  is  transferred  via  shared  memory  to 
the  Collinearity  module,  which  resides  in  the  graphics  engine's 
VME  chassis.  The  i860-based  Collinearity  module  translates 
the  list  of  photocoordinates  into  the  current  estimate  of  head 
location.  For  reasons  explained  in  Section  6.  an  additional 
68030-based  processor  is  used  to  aid  the  transfer  of  dau  from 
the  remote  system  to  the  host.  In  theory,  this  is  not  required. 
The  VME  systems  are  cormected  by  a  Bit-3  VME  buslink. 


The  sampled  head  position  is  communicated  to  the  Pixel-Planes 
5  graphics  engine  [14],  which  in  turn  updates  the  images  on  the 
user  s  displays. 


4  Low-level  software 


3J  Data  Flow 

As  shown  in  Figure  3.  the  signals  emerging  from  the  head- 
mounted  tensors  are  connected  to  the  Remote  Processor.  Worn 
u  a  belt  pack,  the  Remote  Processor  functions  as  a  remote 
analog-to -digital  conversion  module.  It  can  accept  the  four 
analog  voltages  emerging  from  a  lateral-effect  photoJiodc,  for 
up  to  eight  sensors.  On  command,  the  Remote  Processor  will 
simultaneously  sample  the  four  voltages  on  a  selected  sensor 
and  relay  four.  12-bit  results  to  the  LED  Manager.  The  Remote 
Processor  was  used  to  alleviate  the  need  for  long  runs  of  analog 
signals  emerging  from  multiple  sensors. 


A  library  of  low-level  routines  running  on  the  LED  Manager, 
ealled  the  Acquisition  Manager,  controls  the  beacons  and 
detectors.  Given  an  LED  and  a  photodiode  unit,  these  routines 
light  an  LED  and  determine  if  a  photodiode's  detector  secs  that 
LED.  The  dnector  returns  four  analog  signals,  which  the 
Remote  Processor  board  digitizes.  A  simple  formula  [16] 
converts  these  four  numbers  into  the  z.y  photocoordinates  of 
the  LED's  projection  on  the  detector. 

Hamamatsu  datasheets  specify  1  pari  in  40  accuracy  and  1  part 
in  3000  resolution  for  the  lateral-effect  diode. based  detectors 
used  As  with  Antonsson  [2).  we  were  able  to  achieve 


approximately  I  part  in  lUOU  accuracy  for  the  combined 
photodiode- lens  assembly.  Achieving  this  result  required 
significant  efforu  to  improve  the  signal-io-noise  ratio  and 
compensate  for  distortion,  including; 

Ambient  light  rejeeiion:  The  voltage  values  with  the  LED  off 
(called  the  “dark  current*)  are  subtracted  from  the  voltage  values 
with  the  LEO  on.  Sampling  with  the  LEO  off  both  before  and 
after  the  samples  with  tte  LED  on  and  averaging  the  two  yields 
substantially  improved  ambient  light  rejection. 

Handom  noise  rejection;  Averaging  several  measurements 
reduces  random  noise  effecu.  but  costs  time.  A  good 
compromise  between  accuracy  and  sampling  speed  is  to  take  8 
samples  with  the  LED  off.  16  samples  with  the  LEO  on  and  8 
more  samples  with  the  LEO  off. 

Current  scaling:  The  distance  between  a  photodiode  and  an  LED 
depends  on  the  user's  location.  To  maximize  the  signal  without 
saturating  the  photodiode  detector,  the  Acquisition  Manager 
dynamically  adjusts  the  amouru  of  current  used  to  light  an  LED. 
Acquisition  Manager  routines  estimaie  the  threshold  of  current 
that  will  ssiurate  the  detector  attd  use  90%  of  this  value  during 
sampling. 


Figure  6:  Optical  bench  for  photodiode  calibration 


Collation:  Bodi  the  leiu  and  the  photodiode  detector  suffer 
from  nonlinear  distortiora.  By  facing  the  photodiodes  on  an 
optical  bench  and  carefully  measuring  the  imaged  points 
generated  by  beacons  at  known  locations  (Figure  6).  we  built  a 
lookup  uble  to  compensate  for  thr.se  distortions.  Bilinear 
interpolation  provides  compleu  coverage  acrou  the  detector. 
More  sophisticated  calibration  techniques  should  be 
investigate.  Accurate  calibration  is  required  to  reduce  beacon 
switching  error. 

Programming  techniques:  Techniques  such  as  list  processing, 
cache  management  and  efneiem  code  sequencing  result  in  a 
substantially  improved  sampling  rate.  In  addition,  expedited 
handling  of  special  cases,  such  u  when  an  LED  is  not  within 
the  field  of  view  of  a  photodiode  unit,  further  helps  system 
performance. 

Using  32  samples  per  LED.  *c  compute  a  visible  LED's 
phoiocoordinaie  in  660  nscc  and  reject  a  non-visiblv  LED  m 


lOU  jiscc.  LEDs  arc  tested  in  groups;  each  group  carries  an 
additional  overhead  of  60  jiscc. 


Figure  7:  Sensors  viewing  LEDs  in  the  ceiling.  Each  of  the  four 
groups  is  the  set  of  LEDs  that  a  sensor  can  see.  Picture  taken 
with  a  camera  that  is  sensitive  to  infrared  light. 


S  LED  Manager 

The  LED  Manager  uses  the  low-level  Acquisition  Manager 
routines  to  determine  which  LEDs  each  photodiode  unit  sees 
and  where  the  associated  imaged  points  are  on  the  photodiode 
detectors.  We  usually  want  to  collect  data  from  all  visible 
LEDs,  since  larger  sample  seu  ultimately  yield  less  noisy 
aolutiotu  from  the  Collinearity  module  (Section  7).  Because 
the  number  of  visible  LEDs  is  siital)  (see  Figure  7)  compared  to 
the  total  number  of  LEDs  in  the  ceiling,  something  faster  than 
a  brute-force  scan  of  the  entire  ceiling  array  is  called  for.  Two 
assumptions  help  us  design  a  more  efficient  method: 

/>  Spatial  coherence:  The  set  of  beacons  visible  to  a 
photodiode  unit  in  a  given  frame  will  be  contiguous. 

2)  Temporal  coherence:  The  user's  movement  rate  will  be  slow 
compared  to  the  frame  rate.  This  implies  that  the  field  of  view 
of  a  given  photodiode  unit  docs  not  travel  very  far  across  the 
ceiling  between  frames,  so  its  set  of  visible  beacons  will  not 
change  much  from  one  frame  to  the  nexL 

5.1  The  basic  method 

In  each  frame,  the  LED  Manager  goes  through  each  photodiode 
unit  in  sequence,  sampling  beacons  until  it  is  satisfied  that  it 
has  captured  most  of  each  photodiode  unit's  visible  set.  A 
basic  difficulty  is  that  we  caruiot  be  sure  whether  a  beacon  is 
visible  or  not  until  we  suempt  to  sample  it.  The  LED  Manager 
remembers  which  beacons  were  in  the  camera's  visible  set  from 
the  previous  frame.  The  set  is  called  the  last  visible  set.  If  the 
last  visible  set  is  nonempty,  all  beacons  in  that  set  are  tested. 
The  next  action  depends  on  how  many  of  those  beacons  arc 
actually  visible: 

1)  All:  We  assume  the  field  of  view  has  not  moved  much  and 
not  many  more  beacons  will  be  visible.  We  stop  with  this  set 
and  go  on  to  the  next  photodiode  unit. 

2)  Some  'Ve  assume  iha:  the  field  of  view  has  shifted 
significantly,  possibly  cnoui;;i  to  include  previously  unseen 
beacons  A  yhcll  fill  (described  later)  is  conducted,  beginning 
with  the  ^et  of  b«.-.ieons  'erifiej  to  be  Msible 


J)  Sont:  The  Held  of  view  has  moved  dramatically,  gone  off 
the  edge  of  the  ceiling,  or  is  obscured.  We  check  the  neighbors 
of  the  last  visible  set.  If  any  of  these  beacons  are  visible,  they 
are  used  to  start  a  shell  ftll.  If  none  are  visible,  we  give  up  on 
this  photodiode  unit  until  the  next  frame. 

What  if  the  last  visible  set  is  empty?  Our  course  of  action 
depends  on  whether  we  were  able  to  compute  a  valid  position 
and  orientation  for  the  head  in  the  last  frame: 

1)  Valid  prtvious  location;  We  can  predict  which  LEDs  should 
be  visible  to  our  photodiode  unit,  if  the  user's  head  is  actually 
at  the  eonputad  location,  because  the  geometry  of  the  head  unit 
is  known.  If  no  LEDs  are  predicted  to  be  visible,  we  go  on  to 
'iie  next  photodiode  unit,  otherwise  we  sample  those  beacons 
and  use  them  as  the  start  of  a  shell  fiU.  if  any  of  them  were 
actually  visible. 

2)  No  valid  previous  location:  Now  we  have  no  way  to  guess 
which  beacons  are  visible,  so  we  resort  to  a  simple  sweep 
search,  which  lights  the  beacons  in  the  ceiling  row  by  row, 
until  we  have  tried  the  entire  ceiling  or  an  LED  is  found  to  be 
visible.  In  the  former  case,  we  give  up.  and  in  the  latter  case, 
we  use  the  visible  beacon  as  the  start  of  a  shell  fill. 

5.2  Shell  fill 

A  sheU  fill  starts  with  a  set  of  beacons  known  to  be  visible  to  a 
sensor  and  sweeps  outward  until  it  has  found  all  the  beacons  in 
the  field  of  view. 

We  do  this  by  first  sampling  the  neighbors  of  the  initial  set  of 
beacons.  Jf  none  are  found  visible,  the  shell  fill  terminates, 
concluding  that  the  beacons  in  the  initial  set  are  the  only 
visible  ones.  If  any  are  found  visible,  we  then  compute  the 
neighbors  of  the  beacons  we  just  sampled,  excluding  those 
which  have  already  been  tried,  and  sample  those.  We  repeat 
this  process  of  sampling  beacons,  computing  the  neighbors  of 
those  found  visible,  and  using  those  neighbors  u  the  next 
sample  set,  until  an  iteration  yields  no  additional  visible 
beacons. 

Assumption  1,  that  visible  seo  are  contiguous,  suggests  that 
this  procedure  should  be  thorough  and  reasonably  effideni. 

SJ  Startup 

At  stamp,  the  head  location  is  not  known  and  all  of  the  last 
visible  sets  are  empty.  We  do  a  sweep  search,  as  previously 
described,  for  each  photodiode  unit  to  locate  the  initial  visible 

sets. 


i  Communications 

Communication  between  the  various  processors  in  our  system 
is  done  using  shared  memory  buffers,  which  offer  low  latency 
and  high  speed.  TIm  buffen  are  allocated  and  deallocated  via  a 
FIFO  queue  mechanism.  Data  is  "transmitted*  when  it  is  written 
to  the  buffer:  no  copying  is  necessary.  The  only 
communication  overhead  is  the  execution  of  a  simple 
semaphore  acquisition  and  pointer  management  routine. 
Furthermore,  all  processon  use  the  same  byte  ordering  and  data 
type  size,  so  no  data  uanslation  is  needed. 


The  queuing  mechanism  lets  all  modules  in  the  system  run 
asynchronously.  LED  Manager,  the  Collineariiy  module,  and 
Pixel-Planes  5  run  as  fast  as  they  can.  using  the  most  recent 
data  in  the  queue  or  the  last  known  data  if  the  queue  is  empty. 

The  various  processors  in  our  system  are  split  between  two 
separate  VME  buses,  which  are  transparently  linked  together 
by  Bit*3  bus  link  adapters  (Figure  5).  A  subtle  bus  loading 
problem  prevents  the  i860  board  and  the  '030  board  that  runs 
LED  Manager  from  operating  in  the  same  VME  cage.  This 
configuration  increases  latency  because  inter-bus  access  is 
significantly  slower  than  intra-bus  access,  but  increases 
th^ghput  because  the  bus  link  allows  simultaneous  intra-bus 
activity  u>  occur.  Because  the  i860  processor  caiuiot  directly 
acceu  the  VME  bus.  a  second  *030  boartl,  which  runs  the  (Jueue 
Manager,  moves  data  between  the  LED  Manager  and  the 
CoUinearity  module. 

A  simpler  and  less  expensive  system  could  be  built  if  we 
acquir^  an  i860  board  that  can  run  on  the  same  bus  as  the  LED 
Manager  ‘030  board.  This  configuration  would  not  require  the 
Queue  Manager  board  or  the  Bit-3  links  and  would  reduce  both 
latency  and  throughput. 

7  Space  Resection  by  CoUinearity 

Given  the  observations  of  beacons,  we  compute  the  position 
and  orienution  of  the  user's  head  by  using  a  photogrammetric 
technique  called  space  resection  by  collinearity.  ‘The  basic 
method  for  a  single  camera  is  in  [31];  what  we  describe  here  is 
our  extension  for  using  it  in  a  multi-sensor  system.  Because  of 
space  limiutions,  the  description  is  necessarily  brief.  Full 
details  are  provided  in  [6]. 

7.1  DcflaltloBs 

Three  types  of  coordinate  systems  exist:  one  World  space  (tied 
to  the  ceiling  structure),  one  Head  space  (tied  to  the  HMD),  and 
several  Photodiode  spaces  (one  for  each  photodiode  unit). 


Figure  8.-  World,  Head  and  Photodiode  spaces 


Changing  representations  fiom  one  space  to  another  is  done  by 
a  rotation  followed  by  a  translation.  We  use  two  types  of  3x3 
rotation  matrices: 

M  ■  Head  space  to  World  space 
M,  ■  Photodiode  space  i  u>  Head  space 


with  each  matrix  specined  by  Euler  angles  to.  a,  and  r. 


LED;  jf 


The  optical  model  for  each  photodiode  uitii  is  simple:  a  light 
ray  suikes  the  front  principal  point  and  leaves  the  rear 
principal  point  at  the  same  angle  (Figure  9). 


Photodiode  unit  i 


Figure  9:  OptieeU  model 

Rnally.  we  list  the  points  and  vectors  we  will  need,  segregated 
by  the  coordinate  system  in  which  they  are  represented.  Given 
photodiode  unit  i  sees  LED  number  j. 

Photodiode  space; 

0]  ■  imaged  point  on  photodiode  detector 


Figure  10:  Ei^ressing  Ty  through  other  vectors 


Headspace: 

•  vector  from  rear  principal  point  to  imaged  point 
He  m  origin  of  Head  space 

d;  •  vector  from  Ho  to  center  of  photodiode  detector 
e,'  •  vector  from  Hg  to  rear  principal  point 
fj  ■  vector  from  Ho  to  front  principal  point 


Substituting  (2)  and  (3)  into  (1)  yields  the  collineariry 
condition  equation  Cy- : 


’  Xo  X;  ■ 

1 

'■*.7' 

Ye-Yi 

fM  di-tiwMi 

ZSft  *  2|'  . 

\ 

Lo  J 

World  space; 

(Xo,  Xp.  Zo]  •  coordinates  of  the  origin  of  Head  space 
[Xf,  Yj.  Zj]  *  coordinates  of  LED; 

Tjf  m  vector  from  LED  ;  to  front  principal  point 

7.3  Geometric  relationships 

Rgure  9  shows  that  and  t^  differ  only  by  a  scale  factor,  if 
they  were  fdaced  at  the  tame  start  point,  they  would  be 
collinear.  In  equations: 

T*-XMt.,  (1) 

We  now  express  and  t^  in  teims  of  the  other  vccton  in 
aquations  (2)  and  (3)  and  Figures  10  and  11: 


Ti/. 


XoXj 
Ye  -  Yi 
ZeZi 


f  M  r, 


■d. 


(2) 


(3) 


7J  System  of  equations 

When  a  photodiode  unit  i  sees  an  LED  ;,  it  generates  a  c,;.  which 
represents  three  independent  equations.  If  we  see  ^  LEDs  in  all. 
(be  total  number  of  unknowns  in  our  system  is  6-^Af:  3  for 
position,  3  for  orienution,  and  N  scale  factors.  The  first  six 
am  what  we  are  trying  to  find,  but  we  do  not  care  about  the  scale 
factors.  We  eliminate  these  by  reairanging  the  c,y  equations, 
then  dividing  the  fint  and  second  equations  by  the  third.  This 
leaves  two  independent  equations,  of  the  form 

Cf^L)«0.  C2^L)-0 


where  L  is  a  vector  composed  of  the  six  unknowns:  position 
(Xp,  Yg,  Zg)  and  orientation  (at,  a,  ir  for  matrix  M ).  We 
generate  a  linear  approximation  to  these  two  equations  by 
flying  Taylor's  theorem: 


-C1(<L)- 


/dCl«(L) 


\  hYe  I 


dZo 


\  dXo 
jBClyd)' 
do) 

and  a  similar  expansion  for  the  linearized  C2  equation. 


lU.  (!£!2«dL„ 

\  da  /  {dr 


dK 


Now  we  have  six  total  unknowns,  and  every  LED  that  wc  see 
generates  two  independent  linear  equations.  Thus,  wc  need  tu 
see  at  least  three  LEDs.  If  wc  sec  a  loul  of  N  LEDs,  we  can  wtiil' 


our  system  of  N  linetfized  Cl  equations  and  N  linearized  G2 
aquations  in  maths  fonn: 

-Co  -  dC  •  D  (4) 

2^xl  Wx6  6x1 

where  D  « [dXo, dY(^ iZo,  dot,  da,  dk\^. 

dG  is  the  matrix  of  partial  dehvatives  of  the  Cl  end  C2. 

md  -Gq  contains  the  values  of  the  Gf  and  G2  at  a  speahe  L. 

7.4  Iteration  and  convergence 

CoUincahty  takes  an  initial  guess  of  L  (the  unknowns)  and 
generates  oorToetka  values  (in  D)  to  make  a  more  accurate  L. 
iterating  until  it  converges  to  a  solution.  Thus,  we  need  to 
extract  D  from  equation  (4).  If  Af «  3,  then  we  can  solve  for  D 
directly.  If  Af  >  3.  then  the  system  is  overdetermined  and  we 
approximate  D  through  singular  value  decomposition  (24], 
Simulations  show  that  using  more  than  the  minimum  of  3  LEDs 
can  reduce  average  error  caused  by  non>systematic  error 
sources.  In  pseudo^e.  our  main  loop  is; 

Generate  an  initial  guess  for  L 
repeat 

Given  L,  compute  Gq  and  9G 

Estimate  D  tuing  singular  value  decorrqiosiiion 

L-Lf  D 

until  magnitude  of  D  it  small 
return  L 

How  do  we  generate  the  initial  guess  of  L?  Normally  we  use  the 
last  known  position  and  ohentation,  which  should  be  an 
excellent  guess  because  we  track  at  rates  up  to  100  Hz. 
CoUineahty  usually  converges  in  1  or  2  iteratioru  when  the 
guess  is  close.  But  in  degenerate  cases  (at  system  startup,  or 
when  we  lose  iracldng  because  the  photodiode  units  are  pointed 
away  from  the  ceiling},  we  have  no  previous  L.  CoUiaeanty 
will  not  converge  if  the  guess  is  not  close  enough  to  the  true 
value;  we  empirically  found  that  being  within  30”  snd  several 
feet  of  the  true  L  is  a  good  rule  of  thumb.  So  in  degenerate 
cases,  we  draw  initial  guesses  for  L  from  a  precomputed  lookup 
table  with  120  enuies.  trying  them  sequentially  until  one 
converges.  We  can  double-check  a  result  that  converges  by 
comparing  the  set  of  LEDs  used  to  generate  that  solution  to  the 
theoretical  set  of  LEDs  that  the  photodiode  units  should  see.  if 
the  head  actually  wu  at  the  location  just  computed.  When 
these  two  sets  match,  we  have  a  valid  solution. 


S  Performance 

A  'typical  situation'  is  defined  u  a  user  of  average  height 
standing  erect  underneath  the  ceiling,  with  at  least  three 
photodiode  uniu  aimed  at  the  ceiling,  moving  his  head  si 
moderate  speeds.  All  measurement  bounds  assuttK  that  the  user 
remaitu  in  tracker  range  with  at  least  two  sensors  aimed  at  the 
ceiling. 

Updatt  rat*:  The  update  rate  ranges  between  20-100  Hz.  Under 
typical  situations,  50-70  Hz  is  normal,  depending  on  the 
height  of  the  user.  The  wide  variation  in  the  .lumber  of  LEDs 
seen  by  the  sensors  causes  the  variation  in  update  rate.  The 
more  LEDs  used,  the  slower  the  update  rate,  because  LED 
Manager  is  the  slowest  step  in  the  pipeline.  If  the  head 
remains  still  and  the  tensors  sec  a  total  of  B  beacons,  LED 


Manager  requires  3.33  0.782*fl  ms  to  run.  Rapidly  rotating 

the  head  increases  this  lime  by  a  factor  of  about  1.33.  since 
additional  time  is  required  to  handle  the  changing  working  sets 
of  LEDs.  Slower  head  movement  rates  have  correspondingly 
smaller  factors. 

Lag:  Lag  varies  between  20-60  ms.  with  30  ms  being  normal 
under  typical  situations.  Lag  is  measured  from  the  time  that 
LED  Manager  starts  to  the  time  when  the  Collinearity  module 
provides  a  computed  head  location  to  the  graphics  engine. 
Therefore,  tracker  latency  is  a  fimetion  of  the  number  of  LEDs 
seen  and  the  quality  of  the  initial  guess  provided  to  the 
Collinearity  module.  As  B  gets  smaller,  both  the  LED  Manager 
and  Collinearity  modules  become  faster,  reducing  latency.  This 
mutual  dependence  on  B  means  that  update  rate  and  lag  are 
closely  tied:  faster  update  rates  correspond  with  lower  latency 
values. 

Rtsolution:  When  moving  the  head  unit  very  slowly,  we 
observed  a  resolution  of  2  mm  in  position  and  0.2  degrees  in 
orientation.  Measuring  accuracy  is  much  harder,  and  we  do  not 
have  any  firm  numbers  for  that  yeL  At  SIGGRAPH  '91.  users 
were  able  to  touch  a  chair  and  the  four  ceiling  support  poles 
based  solely  on  the  images  they  saw  of  models  of  the  chair  and 
the  poles  in  the  virtual  enviroiunent. 


9  Evaluation 

The  system  provides  adequate  performance  but  has  several 
limitatioru  and  problems  that  must  be  addressed.  The  most 
noticeable  is  the  combination  of  excessive  head-bom  weight 
and  limited  head  rotation  range.  Rotation  range  depends 
heavily  on  the  user's  height  and  position  under  the  ceiling.  A 
typieal  maximum  ^tch  range  near  the  center  of  the  ceiVuig  is 
45  degrees  forward  and  45  degrees  back.  When  the  user  walks 
near  an  edge  of  the  ceiling,  h^  rotation  range  becomes  much 
more  restricted.  To  accommodate  the  full  range  of  bead  motion, 
multiple  image  sensors  must  be  oriented  such  that  wherever  the 
head  is  pointed,  two  or  more  sensors  are  able  to  view  LEDs  on 
the  ceiling.  Given  the  current  focal  lengths,  simulations  show 
that  as  many  as  eight  fields  of  view  are  required  for  a 
respectable  rotation  range  [29].  The  weight  of  each  sensor 
must  be  significantly  reduced  to  achieve  this  goal 

To  reduce  weight,  we  are  trying  to  replace  the  current  lenses  (1 1 
oz.  each)  with  smaller,  lighter  lenses  (2  oz.  each).  Other 
approaches  are  possible.  Wang  proposed  optically 
multiplexing  multiple  fields  of  view  onto  on  a  single  lateral- 
effect  photodiode  [29].  Reduced  signal  strength,  distortions, 
and  view  identification  ambiguities  make  this  a  nontrivial 
task.  It  may  be  easier  to  design  a  helmet  with  integral 
photodiodes  and  lenses.  Given  that  each  photodiode  is  about 
the  size  of  a  quaner,  the  entire  surface  of  a  helmet  could  be 
studded  with  sensors. 

Beacon  switching  error  has  been  greatly  reduced,  but  not 
eliminated.  Small  observable  discontinuities  occasionally 
occur,  and  while  they  are  not  a  major  disturbance,  they  arc 
annoying.  Calibration  techniques  are  being  explored  to 
estimate  error  sources  and  compensate  for  their  effects. 
Photogrammctric  techniques  like  the  bundle  adjustment  method 
(8)  or  an  alternate  scheme  suggested  by  our  colleagues  [18]  may 
provide  the  answer. 


Infrared  light  sources  in  the  environment  surrounding  the 
tracker,  such  as  sunlight  or  incandescent  light,  must  be 
controlled  for  the  system  to  operate  correctly.  Specifically, 
any  light  source  whose  wavelengths  include  880  nm  will  be 
detected  by  the  photodiodes  as  if  it  were  an  LED.  For  this 
reason,  fluorescent  ambient  lighting  is  preferred.  Extreme 
caution  is  not  required,  however.  Whereas  a  sensor  pointed 
directly  at  an  infrared  light  source  other  than  the  LEDs  will 
confuse  the  system,  a  certain  level  of  indirect  infrared 
background  light  is  tolerable  due  to  the  combination  of  optical 
fillers  and  the  ambient  light  rejection  techniques  described  in 
Section  4. 

Surprisingly,  the  bottleneck  in  the  system  is  the  time  required 
to  extract  dau  from  the  photodiode  detectors,  not  the  time 
required  to  compute  the  hud's  location.  The  i860  processor 
perfotnu  the  latter  task  adequately,  and  even  futer  and  chuper 
processors  will  be  available  in  the  future.  But  getting  accurate 
photocoordinates  from  the  detectors  takes  longer  than 
expected,  because  of  the  time  spent  in  cunent  sealing  and  in 
sampling  muluple  time.;  per  LED.  Further  experimentation  is 
required  to  su  if  we  can  safely  reduce  the  number  of  samples. 
Optimizing  the  low-level  software  may  improve  sampling 
speed  by  20-30%. 

The  use  of  Euler  angles  in  the  coUinearity  equations  opens  the 
possibility  of  gimbal  lock.  The  current  system  avoids  this 
because  the  head  roution  range  is  too  limited  to  reach  gimbal 
lock  positions,  but  a  future  version  may.  If  we  cmnot  place  the 
gimbal  lock  positioiu  out  of  reach,  we  can  solve  for  the  nine 
roution  matrix  parameters  individually,  subject  to  six 
cofuiraints  that  keep  the  matrix  special  orthogonal,  or  we  may 
be  able  to  recast  the  rotations  u  quaternions. 

Since  this  tracker  encouregu  the  user  to  walk  around  large 
tpecu,  tripping  over  the  supporting  cables  is  a  danger.  We 
will  investigate  the  feasibility  of  a  wireless  daulink  to  remove 
this  problem. 

Under  certain  circumstances,  the  sensors  can  see  large  numbers 
of  beacons,  such  u  a  total  of  30  or  more.  While  using  many 
LEDs  usually  improves  the  solution  from  the  CoUinearity 
module,  it  alM  slows  down  the  updau  rau  and  increases  the 
teg.  Further  experiments  are  needed  to  explore  this  tradeoff  nd 
determine  rules  of  thumb  that  provide  a  reasonable  balance 
between  resolution  and  update  rate. 

Cellular  systems  using  different  technologies  or  conriguraiions 
could  be  built  to  achieve  similar  scalable  work  areas.  For 
example.  Ascension  has  announced  a  cellulw  magnetic  system 
(4].  Regardless  of  the  technology,  any  cellular  approach 
creates  the  problem  of  beacon  switching  error  or  its  equivalent. 
Steps  we  took  to  control  these  errors  would  apply  to  other 
technologies  u  well:  1)  precise  positioning  and  measurement 
of  system  components.  2)  averaging  techniques  to  reduce 
random  error  sources,  and  3)  calibration  routines  to  compensate 
lor  systcmstic  error  sources. 


10  Future  work 

Wc  intend  lo  continue  improving  this  system.  In  addition  to 
the  tasks  listed  in  Section  9.  wc  would  eventually  like  to 


expand  the  ceiling  size  to  around  20'  x  20'.  to  provide  much 
greater  range  of  movement,  both  quantitatively  and 
psychologically.  Also,  ample  room  exists  to  improve  the 
heuristics  and  optimize  the  code,  increasing  the  update  rate  and 
reducing  latency. 

But  beyond  these  incremental  improvements,  we  do  not  expect 
to  pursue  this  particular  technology  further.  The  system  is  a 
vehicle  for  further  research  and  provides  room-sized  tracking 
capability  today  for  HMD  applications  that  require  it.  For 
example,  the  UNC  Walkthrough  team  has  begun  interview- 
based  user  studies  on  what  impact  large -environment  tracking 
has  on  the  architectural  design  of  a  kitchen.  In  the  future, 
emphasis  will  be  placed  on  technologies  that  allow  unlimited 
tracking  volumes  in  unstructured  environments.  This  potential 
exists  in  systems  that  measure  only  the  relative  differences  in 
position  and  orientation  as  the  user  moves,  integrating  these 
difrerences  over  time  to  recover  the  user's  location.  Examples 
include  inertial  technologies  and  Self-Tracker.  Since  these 
technologies  suffer  from  drift  problems,  initial  versions  may 
be  hybrid  systems  reliant  on  the  optical  tracker  for  auxiliary 
information.  Thus,  the  optical  tracldng  system  will  serve  as  a 
teatbed  for  its  own  successor. 

Tracking  HMDs  will  only  get  harder  in  the  future.  The  higher 
resolution  displays  being  developed  demand  higher  resolution 
trackers.  See-through  HMDs  sdd  additional  requirements.  In 
the  completely-enclosed  HMDs  commonly  used  today,  the 
entire  world  is  virtual,  so  resolution  is  much  more  important 
than  accuracy.  But  for  s  aee-through  HMD.  accurate 
registration  of  the  HMD  to  the  real  world  is  vital.  The  effects 
of  latency  will  alto  become  more  disturbing  in  see-through 
HMDs.  Viewing  computer-generated  objects  superimposed 
upon  the  real  world,  wh^e  those  objecu  move  with  significant 
Ul  but  the  real  world  does  not.  will  not  provide  a  convincing 
illusion.  People  can  perceive  as  little  as  5  ms  of  lag  [IS],  and 
it  is  unlikely  that  the  combined  tracker  end  graphics  engine 
latency  will  be  below  that  anytime  soon.  Therefore, 
compensation  techniques  need  to  be  explored  119}[24].  If 
HMDs  are  to  achieve  their  potential  of  making  a  user  truly  feel 
immersed  inside  a  virtual  world,  significant  advances  in 
tracking  technologies  must  occur. 
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Walkthrough  is  a  system  to  help  architects  and  their  clients  explore  a  building  design  prior  to  its  con¬ 
struction,  correcting  problems  on  the  computer  instead  of  in  concrete.  To  sustain  the  illusion  of  being 
inside  a  virtual  building,  we  must  make  it  possible  to  move  through  the  building  in  a  natural  manner, 
with  as  few  encumbrances  as  possible.  The  head-mounted  display  ana  optical  tracker  allow  a  large 
working  volume  where  the  user  can  walk  naturally  in  the  virtual  building.  Goals  of  optical  tracker 
research  include  using  see-through  optics  in  the  head-mounted  display  and  tracking  in  unstructured 
environments,  such  as  an  outdoor  building  site. 


Instructions 

Use  these  guidelines  to  help  you  experience  the 
Optical  Tracker  demonstration.  The  goal  of  the 
demonstration  is  to  allow  you  to  fully  experience  a 
virtual  world  application  in  the  limited  time  avail¬ 
able. 

1)  Sit  in  the  special  chair  that  is  bolted  to  the 
floor.  A  UNC  assistant  will  place  the  head- 
mounted  display  on  your  head  and  will  remove 
it  after  the  demonstration.  The  headgear  is  very 
heavy  (10  lbs).  You  may  want  to  hold  the  VPL 
EyePhone  portion  to  steady  it  during  the 
demonstration.  Please  do  not  touch  the  rest  of 
the  head-mounted  display;  it  is  a  prototype 
and  is  liable  to  damage. 

2)  Be  careful  to  notice  the  limits  of  motion.  The 
tracker  does  not  track  as  well  near  the  edges  of 
the  ceiling,  because  one  or  more  sensors  may 
be  off  the  ceiling.  The  ceiling  is  about  the  size 
of  the  models,  so  when  you  are  near  the  walls 
of  the  model  you  are  near  the  edge  of  the 
ceiling. 

3)  Walk' 

4)  At  the  end  of  the  3-minute  demo,  walk  back  to 
the  chair  and  try  to  sit  back  down  in  it.  The 


Figure  1.  With  “inside-our  tracking,  multiple  head- 
mounted  image  seraon  view  LED  beacons  suspended  in 
ceiling  panels.  In  this  drawing,  each  sensor’s  field  of  xhew  is. 
mapped  onto  the  ceiling  to  show  that  more  than  three  LEDs 
are  typically  visible. 


chair  exists  in  the  computer  model  of  the 
simulated  environment.  Use  the  computer- 
generated  images  you  see  in  the  head-mounted 
display  to  guide  you. 
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The  Tracker 

An  optoelectronic  tracking  system  that  adopts  an 
inside-out  paradigm  introduced  by  Gary  Bishop 
and  Henry  Fuchs  in  1984  is  used  in  the  demonstra¬ 
tion.  The  current  prototype  features  2-x  2-foot 
ceiling  panels  in  a  10-x  12-foot  room  with  32 
individually  addressable  light  emitting  diodes 
(LEDs)  mounted  in  each  panel.  The  LEDs  serve  as 
a  global  position  reference  for  a  head-mounted 
display  that  is  augmented  with  four,  upward 
looking  image  sensors.  Each  sensor  resembles  a 
small  camera  in  that  its  lens  focuses  infrared  light 
onto  a  lateral-effect  photodiode’s  surface.  The 
photodiode,  in  turn,  reports  the  photocoordinates 
of  an  LED  that  has  been  momentarily  turned  on. 
The  photocoordinates  associated  with  three  or 
more  LEDs,  along  with  their  absolute  position  in 
the  room,  allow  the  position  and  orientation  of  the 
head-mounted  display  to  be  computed.  The  sample 
rate  of  the  optical  tracixr  is  dependent  on  the 
number  of  LEDs  imaged  and  ranges  from  35  to 
100  Hz.  A  30  to  50  ms  lag  in  the  calculated  mea¬ 
surements  is  caused  by  a  two  'tage  pipeline  in  the 
architecture. 

The  advantage  of  this  approach  lies  in  its  cellular 
nature.  Typically,  optical  tracking  is  done  with 
fixed  cameras  and  moving  LEDs.  That  is,  the 
cameras  are  outside  looking  “in.”  A  fundamental 
limitation  exists  with  this  approach:  the  working 
volume  is  limited  to  the  cameras’  field  of  view. 
Furthermore,  to  achieve  high  resolution,  long  focal 
lengths  are  required  and  a  reduction  in  working 
volume  follows.  Reversing  the  situation,  i.e., 
moving  the  cam.eras  and  fixing  multiple  LEDs, 
solves  this  dilemma  Long  focal  lengths  can  be 


Figure  2.  Floor  plan  of  the  living  room  of  a  hypothetical 
apartment  model. 


used  to  achieve  high  resolution  and,  as  the  cameras 
move,  the  set  of  LEDs  in  a  camera’s  field  of  view 
is  constantly  changing.  This  unique  concept, 
diagrammed  in  Figure  1,  is  the  first  step  in  a  long 
range  research  effon.  Future  goals  will  involve 
reducing  the  number  of  beacons  required  by 
utilizing  sensors  that  are  capable  of  measuring 
incremental  head  motion.  Both  optical  and  inertial 
technologies  are  being  investigated. 

The  ultimate  goal  in  user  interface  should  be  to 
make  the  interface  as  transparent  as  possible  to  the 
user.  Interaction  with  the  model  should  be  as  easy 
and  intuitive  as  interacting  with  the  real  object. 
Thus,  the  devices  should  encourage  interaction 
with  the  model  in  a  natural  manner.  In  this  the 
optical  tracker  hopes  to  excel. 
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3dni  is  a  program  that  allows  the  creation  of  a  variety  of  virtual  geometric  objects.  You  can  fly  around 
in  this  virtual  world  in  which  the  other  user,  the  designer,  is  creating  and  modifying  objects  according  lo 
your  verbal  instructions.  A  variety  of  interesting  things  can  be  created  rapidly  with  3dm,  especially 
things  composed  of  boxes,  spheres,  and  cylinders.  You  can  ask  the  modeler  to: 

■  change  the  color  of  an  object.  For  example,  “make  that  sofa  green.” 

■  copy  and  paste  objects  to  make  multiple  copies.  For  example,  “make  a  few  copies  of  that  tree.’ 

■  make  an  object  bigger  or  smaller  or  reshape  pan  of  an  objecL  For  example,  “make  the  tree 
bigger”  or  “make  that  branch  point  down  more.” 

■  create  objects  like  the  following:  the  Old  Well,  a  tree,  a  poned  plant,  a  rock,  mountains,  sky¬ 
scrapers,  a  water  tower,  a  simple  house,  a  castle,  a  sofa  or  chair,  a  table,  a  fish,  a  jet  or  rocket. 


Instructions 

Use  these  guidelines  to  help  you  experience  the  3D 
modeling  system.  The  goal  of  the  demonstration  is 
to  allow  you  to  fully  experience  a  vinual  world 
scenario  in  the  limited  time  available. 

1 )  Let  the  UNC  assistant  help  put  the  VPL 
EyePhone  on  your  head.  This  is  done  by 
separating  the  earphones  on  the  EyePhone 
while  you  slip  on  the  headgear.  Make  sure  that 
the  EyePhone  fits  snugly. 

2)  You  will  be  holding  a  3D  mouse  with  two 
bunons  mounted  on  it.  In  the  virtual  world  this 
mouse  will  appear  as  a  cursor  in  the  shape  of  a 
small  airplane.  For  best  results,  hold  your  hand 
at  least  6  inches  from  the  EyePhone.  Holding 
your  hand  too  close  to  the  EyePhone  will 
interfere  with  the  Polhemus  tracking  sensor 
and  your  hand  or  the  airplane  will  shake. 

3)  You  will  notice  that  a  yellow  ring  surrounds 
you  on  the  floor.  This  represents  the  range  of 
your  Polhemus  tracker — if  you  walk  beyond  it 
(or  close  to  its  edge),  Polhemus  can  no  longer 
track  your  position  accurately  and  the  image 


that  you  see  of  your  virtual  world  will  shake 
violently.  Walk  back  within  the  ring  to  restore 
normal  tracking.  For  best  tracking  response, 
stand  near  the  Polhemus  tracking  source, 
which  hangs  overhead. 

4)  The  modeler  in  the  virtual  world  is  surrounded 
by  a  red  ring  on  the  floor,  which  represents  his 
or  her  tracking  range.  The  modeler’s  head  and 
body  will  appear  in  red.  You  will  be  able  to  see 
the  modeler’s  cursor,  which  will  take  on 
different  shapes  and  sizes.  The  modeler  also 
has  a  toolbox  with  tools  that  allow  him  or  her 
to  create  objects.  You  will  not  be  able  to 
choose  tools  from  the  toolbox. 

5)  To  fly  forward  (the  direction  the  airplane  is 
pointing),  hold  down  the  left  mouse  button.  To 
fly  backward,  hold  down  the  right  mouse 
button.  Your  speed  will  slowly  increase  the 
longer  you  hold  down  cither  mouse  button.  To 
stop  flying,  release  the  mouse  button. 

6)  Because  you  will  be  flying  in  the  direction 
your  hand  is  pointing  and  not  in  the  direction 
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View  of  the  3dm  Virtual  World 


(Note  The  Madder's  virtual  body,  hand,  range,  source,  and  toolbox  are  not  shown.  The  various  virtual  models  will  not 
necessarily  appear  as  those  in  this  diagram.) 


you  are  looking,  you  may  look  anywhere  while 
flying  without  affecting  your  motion.  Also, 
remember  that  you  fly  in  the  direction  your 
airplane  points,  so  for  most  accurate  flying  it 
helps  to  “sight  down”  the  airplane  (that  is, 
look  at  it  from  directly  behind  its  tail)  to  see 
exactly  where  it  is  pointing. 

7)  You  will  sec  several  objects  that  have  been 
created  in  the  virtual  world.  You  may  fly 
around,  through,  and  into  these  objects,  or 
anywhere  in  the  virtual  world.  You  can  fly  up 


or  down  and  will  not  fall.  There  is  no  gravity 
in  this  virtual  world.  If  you  fly  too  far  from  the 
naodel,  you  may  get  lost.  Scan  the  virtual  world 
by  looking  around  to  flnd  the  model  so  that 
you  can  fly  back  to  it  Remember  to  look  up 
and  down! 

8)  When  the  virtual  world  freezes  and  the  words 
“Game  Over”  appear,  please  wait  for  the  UNC 
attendant  to  help  you  take  off  the  EyePhone. 
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The  intent  of  this  application  is  to  give  you  an  “atom’s  perspective”  of  an  assortment  of  molecules, 
while  demonstrating  a  method  of  navigating  through  a  large  space  in  a  virtual  world.  You  will  use  a 
head*mounted  display  and  a  hand-held  input  device  to  fly  through  a  virtual  world  containing  the  follow¬ 
ing  giant  molecules: 

■  DHFR  (dihydrofolate  reductase)  and  methotrexate-  DHFR  is  a  protein  molecule  and 
methotrexate  is  a  drug  molecule  used  in  cancer  treatmenu  which  is  known  to  dock  with  DHFR. 
Both  are  represented  with  CPK  (sphere-only)  models. 

■  crambin-  a  protein  molecule,  represented  here  by  three  different  models:  CPK  (spheres  represent 
individual  atoms  colored  according  to  type),  sdck-only  (sticks  represent  the  bonds  between  the 
atoms),  and  ribbon  (the  amino  acid  chain  that  makes  up  the  backbone  of  the  model  is  represented 
as  a  ribbon). 

■  clathrate-  a  highly  energetic  compound  which  is  represented  as  a  baJl-and-stick  model. 


Flying  Through 
Molecules 


Instructions 


Use  the  following  guidelines  to  experience  this 
virtual  worlds  application.  The  goal  of  the  demon¬ 
stration  is  to  allow  you  to  fully  experience  a  virtual 
world  scenario  in  the  limited  time  available. 

1)  Let  the  UNC  assistant  help  put  the  VPL 
EyePhone  on  your  head.  TTiis  is  done  by 
separating  the  earphones  on  the  EyePhone 
while  you  slip  on  the  headgear.  Make  sure  that 
the  EyePhone  fits  snugly. 

2)  To  "fly"  through  the  molecules,  grab  the  3D 
mouse  and  point  your  hand  in  the  direction 
you  want  to  go.  lYess  the  left  button  to  go 
forward;  the  right  bunon  to  go  in  reverse. 

3)  When  you  press  a  button  to  fly  in  either 
direction,  you  will  hear  a  “rocket”  sound. 

This  indicates  that  you  are  flying. 


4)  Your  speed  is  proportional  to  the  amount  of 
distance  you  put  between  your  hand  and  your 
body;  i.e.,  to  fly  faster,  you  hold  the  mouse 
further  away  firom  you  and  press  the  bunon. 
You  can  check  your  speed  by  looking  at  the 
color  of  the  hand  icon,  which  is  in  the  shape  of 
an  arrow.  At  minimum  speed,  the  arrow  is 
blue;  at  maximum  speed,  it  is  red.  The  color 
blends  smoothly  for  all  speeds  in  between. 
When  you  are  not  pressing  any  bunons,  the 
arrow  will  be  white.  Be  sure  to  hold  your 
hand  out  in  front  of  you  so  that  you  can  see 
the  direction  and  color  of  the  arrow. 
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Bird’s  Eye  View  of  Molecuie  Fiythrough  Environment 
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Walkthrough  is  a  system  designed  to  help  architects  and  their  clients  explore  a  building  design  prior  to 
its  construction,  correcting  problems  on  the  computer  instead  of  in  concrete.  Using  a  treadmill  and  a 
head-mounted  display,  the  user  can  navigate  through  a  three-dimensional  virtual  building  and  evaluate 
its  design.  A  treadmill  is  an  appropriate  vehicle  to  use  for  this  task  since  it  both  immerses  the  user  in  the 
virtual  environment  and  allows  him  to  explore  it  in  a  natural  maimer-  by  walking. 


Instructions 

Use  these  guidelines  to  help  you  experience  the 
Interactive  Building  Walktlutjugh  demonstradon. 
The  goal  of  the  demonstradon  is  to  allow  you  to 
fully  experience  a  virtual  world  j^plicadon  in  the 
limited  dme  available. 

1)  Let  the  UNC  assistant  help  put  the  VPL 
EyePhone  on  your  head.  This  is  done  by 
separating  the  earphones  while  you  slip  on  the 
headgear.  Make  sure  that  the  EyePhone  fits 
snugly. 

2)  Movement  through  the  model  is  done  by 
walking  on  the  treadmill.  Leaning  forward 


slightly  and  pushing  against  the  handlebars 
will  make  walking  easier.  Your  speed  is 
controlled  by  the  length  of  your  stride  and  how 
quickly  you  walk. 

3)  To  turn,  turn  the  handlebars  as  you  would  on  a 
bicycle.  As  on  a  bicycle,  if  you  turn  the 
treadmill’s  handleban  sharply,  you  will  turn 
rftaiply  in  the  virtual  world  of  the  building. 

4)  Lights  in  the  model  can  be  turned  on  and  off  by 
the  UNC  assistant.  If  you  want  to  change  the 
lights  in  a  room,  tell  the  assistant  which  ones  to 
change. 
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The  basic  goal  of  radiation  treatment  of  a  cancerous  tumor  is  to  irradiate  the  tumor  with  enough  dosage 
to  kill  the  malignant  cells  without  further  hurting  the  padent  This  demonstration  represents  ongoing 
research  at  UNC  using  state-of-the-art  3D  displays  to  design  radiadon  therapy  treatment  plans. 

Convendonal  radiotherapy  treatment  planning  is  based  on  a  handful  of  2D  X-ray  images  showing 
prospective  “beam’s-eye”  views  of  the  patient’s  anatomy.  Dose  fields -for  those  prospective  beam  con¬ 
figurations  are  completed  only  for  isolated  2D  slices  through  the  patient’s  body.  The  result  is  suboptimal 
conformation  between  the  tumor  volume  and  the  proposed  dose  distribution,  both  of  which  have  com¬ 
plex  3D  structures  but  are  studied  only  in  2D. 

Current  state-of-the-an  3D  treatment  planning  takes  advantage  of  today’s  display  and  computational 
power  to  enable  the  radiotherapist  to  study  the  patient  in  three  dimensions.  Interactive  beam  targeting 
using  a  computerized  model  of  the  patient  (as  opposed  to  using  the  real  patient,  as  in  2D  planning) 
permits  the  radiotherapist  to  explore  a  continuum  of  beam  configurations.  3D  dose  grids  are  now  com¬ 
puted  and  registered  with  the  patient  anatomy  model,  providing  better  comparisons  between  tumor 
volume  and  dose  distribution. 

Head-mounted  displays  may  be  useful  for  targeting  treatment  beams  since  they  allow  more  natural 
steering  and  navigation  through  the  patient’s  anatomy  than  that  allowed  by  fixed  workstation  screens. 


Instructions 

This  demonstration  is  designed  to  give  you  a  feel 
for  how  a  head-mounted  display  might  be  used  for 
the  task  of  targeting  radiation  treatment  beams. 

Use  the  following  guidelines  to  experience  this 
virtual  worlds  application.  The  goal  of  the  demon¬ 
stration  is  to  allow  you  to  fully  experience  a  virtual 
world  scenario  in  the  limited  time  available. 

1 )  Let  the  UNC  assistant  help  put  the  VPL 
EyePhone  on  your  head.  TTiis  is  done  by 
pulling  the  headphones  down  over  your  ears 
while  you  slip  on  the  headgear.  Make  sure  that 
the  EyePhone  fits  snugly. 

2)  You  are  presented  with  a  virtual  patient,  a 
polygonal  model  of  a  human  figure  with  a 


portion  of  the  surface  removed  to  reveal 
yellow  tumors  located  on  the  patient's  kidneys. 
Also  visible  are  the  patient's  liver,  spleen,  renal 
anery,  spine  and  ribs. 

3)  Next  to  the  patient’s  head  are  three  treatment 
beams.  Your  task  is  to  position  these  three 
beams  so  that  they  irradiate  the  tumors  while 
missing  as  much  of  the  sensitive  healthy  tissue 
as  possible.  To  manipulate  a  beam,  place  the 
red  tip  of  the  hand  cursor's  index  finger  into  a 
beam.  A  wireframe  bounding  box  appears 
around  the  beam,  indicating  that  the  beam  has 
been  selected.  A  low-frequency  hum  is  also 
audible.  To  grab  the  beam,  press  either  mouse 
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button  while  the  beam  is  selected.  As  long  as 
you  bold  the  button  down  (i.e.  your  hand  is 
“holding”  the  beam),  the  beam  is  controlled  by 
the  mouse.  When  you  release  the  mouse 
button,  the  beam  stays  in  its  current  position. 

4)  Grab  each  beam  and  position  it  so  that  it  passes 
through  the  tumor,  but  hits  as  little  of  the  other 
anatomical  structures  as  possible.  The  multiple 
beams  allow  you  to  attack  a  particular  tumor 
from  different  directions,  so  that  the  niTtyw 
multiply  irradiated  while  healthy  tissue  is  hit 
by  just  one  beaza 

5)  Because  of  the  limitations  of  the  treatment 
machine,  which  is  ultimately  used  to  deliver 
the  beams  you  are  targeting,  not  all  beam 
directions  are  valid.  When  your  beam  is  in  an 
invalid  configuration  it  will  be  colored  cyan, 
whereas  a  valid  beam  configuration  is  colored 
red.  You  have  3  1/2  minutes  to  target  the  three 
beams. 

6)  When  you  hear  the  sound  of  the  harp  glissando, 

your  time  is  up.  You  then  see  your  treatment 
plan  being  administered  to  the  patient.  The 
treatment  machine  appears.  It  and  the  patient 
arc  scaled  down  by  50  percent  to  give  you  a 


Bird's  Eye  View  of  Virtuai  Worid 


better  view  of  the  whole  setup.  Step  back  and 
watch  as  the  table  and  gantry  move  to  deliver 
the  treatment  plan  you  have  specified. 


Credits 

Principal  Investigators:  Frederick  P.  Brooks,  Jr.,  Henry  Fuchs 

Authors:  James  C.  Chung,  Bradley  A.  Crittenden,  Suresh  Balu,  Terry  Yoo 

Data  Courtesy  of:  Dr.  Ron  Kikinis;  Dr.  Ferenc  A.  Jolesz;  Steven  Seltzer,  and  Stuart  Silverman, 
Brigham  &  Women’s  Hospital  and  Harvard  Medical  School,  Boston, 

Bill  Lorensen;  Harvey  Qin,  and  Jon  Zarge,  GE  Corporate  R&D, 

VPL  Research,  Inc. 
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National  Science  Foundation,  Office  of  Naval  Research 
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The  nx>untain  bike  virtual  wealds  applicadon  was  OTiginally  designed  to  be  an  enhanced  version  of  a 
stationary  bicycle,  providing  both  interactive  entertainment  and  exercise  for  the  user.  In  this  scaled- 
down  version,  you  can  "ride"  a  stationary  bicycle  over  terrain  in  a  virtual  world.  The  bicycle  is  equipped 
with  pedal  force  feedback,  which  makes  pedalling  uphill  more  difficult  than  pedalling  downhill,  just  as 
in  the  real  world. 


Instructions 

Use  the  following  guidelines  to  experience  this 
virtual  worlds  application.  The  goal  of  the  demon¬ 
stration  is  to  allow  you  to  fully  experience  a  virtual 
world  scenario  in  the  limited  time  available. 

1)  Seat  yourself  on  the  bicycle. 

2)  Let  the  UNC  assistant  help  put  the  VPL 
EyePhone  on  your  head.  This  is  done  by 
separating  the  earphones  on  the  EyePhone 
while  you  slip  on  the  headgear.  Make  sure  that 
the  EyePhone  fits  snugly. 

3)  When  the  demonstration  begins,  you  will  be 
overlooking  a  textured  terrain.  There  is  a  road 
that  designates  a  path  through  the  world.  If  you 
ride  along  it,  you  will  be  able  to  see  most  of 


the  signs,  buildings  and  other  interesting 
objects  that  are  pan  of  this  virtual  world.  You 
do  not  have  to  stay  on  the  road  but,  since  the 
demonstration  is  only  three  minutes  long,  you 
will  see  most  of  the  attractions  if  you  ride  on 
the  toad. 

4)  When  your  three  minutes  arc  up,  or  when  you 
have  finished  your  ride,  the  words  "Game 
Over"  will  appear.  Wait  for  the  UNC  attendant 
to  help  you  take  off  the  EyePhone. 

5)  Most  importantly,  enjoy  your  ride  and  happy 
exercising. 


Hardware  details 

This  application  uses  a  VPL  EyePhone  for  its  display  device  and  a  Polhemus  3SPACE  TRACKER  for 
tracking  the  location  and  orientation  of  the  user’s  head.  A  Sun-4  workstation  is  used  to  interface  this 
equipment  to  the  graphics  engine — ^Pixel-Planes  5 — a  custom-made,  massively  parallel  imaging  ma¬ 
chine  developed  at  UNC-Chapel  Hill. 

The  bicycle  has  sensors  mounted  on  its  back  wheel  and  front  fork.  These  sensors  measure  the  user's 
distance  and  direction  changes.  The  number  of  rear  wheel  revolutions  is  used  to  measure  distance  and 
speed,  and  the  turning  of  the  handlebars  determines  the  bicycle’s  direction. 
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Mountain  Bike  Terrain 


The  bicycle  has  an  eddy-cunent  resistance  motor  attached  to  the  back  wheel;  a  CompuTrainer,  Model 
CAT  6000  by  Racennate. 

When  the  rider  is  pedalling  uphill,  pedal  resistance  increases  according  to  the  steepness  of  the  grade.  It 
is  not  possible  to  coast  downhill  in  this  scenario  because  the  nootor  that  is  attached  to  the  bicycle  can 
only  affect  resistance  to  the  wheel’s  turning  and  cannot  itself  help  propel  the  wheel. 

Credits 

Authors:  Erik  Erikson,  Ryutaiou  ^buchi,  Andrei  State,  Russell  Taylor. 

This  applicadon  began  as  a  class  project  by  Ryutarou  Ohbuchi  in  Comp  290  (now  239),  “Exploring 
Virtual  Worlds”  (taught  by  Henry  Fuchs). 

Sponsors:  Defense  Advanced  Research  Projects  Agency,  Office  of  Naval  Research 
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One  of  the  wonderful  things  about  displaying  molecules  is  that  there  is  no  single  correct  way  to  do  it. 
Therefore,  many  different  representations  are  possible,  and  chemists  use  different  ones  depending  on  the 
situation. 

This  application  lets  you  stroll  around  several  different  representations  of  a  protein  molecule,  stick  your 
head  inside  them,  and  even  pick  them  up  and  spin  them  around.  The  molecule  is  called  crambin,  and  is 
represented  here  by  three  different  models: 

■  CPK:  spheres  represent  individual  atoms  colored  according  to  type 

■  stick-only:  sticks  represent  the  bonds  between  the  atoms 

■  ribbon:  the  amino  acid  chain  that  makes  up  the  backbone  of  the  model  is  represented  as  a  ribbon 


Molecule 

Museum 


Instructions 

Use  the  following  guidelines  to  experience  this 
vinual  worlds  application.  The  goal  of  the  demon¬ 
stration  is  to  allow  you  to  fully  experience  a  vinual 
world  scenario  in  the  limited  time  available. 

1 )  Let  the  UNC  assistant  help  put  the  VPL 
EyePhone  on  your  head.  This  is  done  by 
separating  the  earphones  on  the  EyePhone 
while  you  slip  on  the  headgear.  Make  sure  that 
the  EyePhone  fits  snugly. 

2)  Viewing:  To  get  a  bener  look  at  a  model,  just 
walk  up  to  it.  Putting  your  head  inside  the 
model  is  strongly  encouraged  (where  else  can 
you  put  your  head  inside  a  molecule?). 

3)  Grabbing:  To  grab  a  model,  put  your  hand 
(while  holding  the  3D  input  device)  inside  the 
model  and  press  either  button  and  hold  the 
bunon  down.  When  your  hand  is  inside  the 
model,  the  model’s  “bounding  box”  will  be 
drawn  around  it  to  highlight  it;  in  addition, 
you  should  hear  a  low  hum  to  indicate  that  the 


model  can  be  grabbed.  When  you  successfully 
grab  a  model,  you  should  hear  a  canoonish 
“floop”  sound  (only  at  the  instant  you  grab-it 
doesn’t  continue  like  the  hum  does).  For  as 
long  as  you  hold  either  bunon  down,  the 
model  will  stay  attached  to  your  hand,  and  will 
follow  your  hand’s  position  and  orientation. 

4)  Releasing:  To  release  the  model,  just  release 
the  bunons. 

5)  You  may  want  to  try  superimposing  one  or 
more  models  on  each  other  to  see  how  the 
different  representations  “line  up.”  Do  this  by 
grabbing  one  and  putting  it  in  the  same  spot  as 
the  other-you  can  put  all  three  on  top  of  each 
other  if  you  wish.  When  two  or  more  models 
are  piled  up  together  in  this  manner,  you  can 
press  the  button  to  grab  whichever  model  is 
closest  to  the  center  of  your  vinual  hand. 
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Bird’s  Eye  View  of  Moiecule  Museum 


Credits 
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